Category Archives: School

Notes from Day 1 of UCalgary CPSC 585 Winter 2014

This semester I am taking the University of Calgary’s Games Programming course, cpsc 585. Like Last semester’s virus course this course also required an essay.  The games course, has lectures for 8 hours per day for 5 days during our block week. A block week is the week before regular classes begin and is used to provide super-concentrate classes or small unmarked courses. After the lectures we’ll spend the rest of the semester programming a racing game as our project. The entire grade is determined by our progress on the game and our final game.

The course is taught by two games developers so there is lots of interesting discussions. Since in general games is a closed community I’ve tried to not include any comments or tidbits which might be insider knowledge.

You may also notice that this course takes a console and AAA focused look. This does mean they are focused on more mainstream games and direction.

These notes were taken in real time. I’m not sure any one will enjoy reading them but I need to write this else I’ll get bored and stop paying attention. Eight hours of listening is a long time!

Tuesday 2nd:

Prior to this we looked at the finance structure of the typical Developer -> Publisher -> Distributor life cycle. I got the impression that work life at Radical improved once EA took them over. This would explain why studios are still selling themselves to EA despite Maxis, or Westwood, or Dice, or you get the idea.

Brief overview of the games industry:

  • Movie games often start production during the movie’s post-production. Too late to make a good game.
  • Publisher handles IP licensing fees
  • Racing games license cars and tracks
  • Manufactuerer’s take is greater than what an indipendent developer will earn on the same sale.
  • Multi-platform AAA development only budget $20 – 30+ million
  • Typical timeframe is 12 – 24 months. Valve was not mentioned.
  • Expensive trends: High production values, open world, licensing tie-ins, celebrity voice acting
  • Big open world games longer at 3+ year production cycle.
  • Games engineering is failry ad hoc.
  • EA has structed their development well but you cannot engineer fun.
  • EA NBA 2010 cancelled, wasn’t fun. Development went to new studio and given full 2 year dev cycle. Still wasn’t fun. Cannot engineer fun, more will be talked about later.
  • Industry expects upstart working hours but pays for 9-5.
  • Little formal software design.
  • 20-30% of team is programmers, 20-30% of team is scripters who have no formal programming education. Rest is art and game design + managers.
  • No initial requirements are fixed.
  • “make it not suck now” is the impreative
  • Like other software systems: core runtime, tools & pipeline.
  • Codebase needs to be maintainable for next year’s game.

Break taken at 10:25.

Back at 10:35. Topic: Fun.

Question: “What do you like about your favourtite games?”

  • Playing with friends in borderlands
  • Challenging/hard
  • Exploring the world like in fallout 3 <- Me

Why do people play games:?

  • Reward
  • Immersion
  • competition
  • escapoe
  • challenge
  • relazation
  • excitement
  • discovery
  • intelletual stimulation
  • accompliishment
  • variety
  • social interacton

All these and what we mentioned goes into what is fun. Theory of Fun: Fun is the process of pattern matching, or grokking a pattern. We have noticed a pattern and used that to “beat” the system = fun in games. Games are safe places to practice grokking still.

Question: What about do you dislike?

  • repition, grinding
  • hand holding
  • cheesy story lines
  • Insane hard puzzles, in adventure games <- Me

What isn’t fun?:

  • punishment over reward
  • tedium
  • inconsistency, Example: In Journey Down you can replace virgin olive oil with moter oil, but not a herb with a leaf.
  • cheating
  • presentation glitches
  • boredom

What kinds of games do we like or dislike:

  • MOBA
  • Open world
  • Copetitive
  • Hate: driving games and sports
  • Any genre, but like COD: online or puzzle solvers

Game Genres:

  • Action/Adventure
  • horro
  • shooter
  • stealth
  • platformer
  • rhytm/music
  • MMO
  • racing
  • sports
  • role-playing
  • turn-based strategy
  • point-and-click adventure
  • puzzle

Gameplay style:

  • Open world vs Levels
  • Casual vs Hardcore
  • Violent vs non-violent
  • first person vs Third person
  • single player vs multiplayer

So while COD and Far Cry are shooters they target a different audience because far cry is open world and single player while COD is leveled and multiplayer.

Non-violent First-Person-Shooter could sell well if it were easy to make. Good example is Portal but that game was hard to make. Style more iportanted to gaming than other medias. Genres bring expectations and decisions. People have cannot switch easy between genres, unlike other medias like say movies. Without competence in shooters Portal cannot be enjoyed to its full.

Genre will define aspects of the game. Can limit creativity. Example: Kart style racing game: you’ll have things on track to pick up and use.

Sucessful games:

  • solid core mechanics
  • sense of place
  • range of challenges
  • range of abilities used to solve the encounter
  • skill required to use the abilities
  • rewards

If you make a new genre/mechanic you must make sure it is solid. Story driven games must have a defined environment where things occur. Even tetris has a location, original used russia. you’re playing tetris is russia, why? because you have to be playing it somewhere! Tetris example of how challenges range from easy on the first level to hard on later. Shooters: you can take different approaches to solve level. You can sneak or go in guns blazing. New equitment can open up new solutions. Give the player onus for rewards, not “press button -> cool thing happens”. Archem Asylum: last baddy taken out -> camera focuses in and shows fuller detail.

Fun is Hard Work TM

  • Fun more difficult to plan and schedule than other portions of development
  • Often don’t know if it is fun until its built
  • Low tollerance to mistakes
  • Nothing is fun while you’re working on it, must be brought out
  • Exaple: Let’s add cool reflections! -> “Oh, hose reflections do not look good, at all. We’ll have to remove reflections as a mechanic in that one level”

Coure-meta; marks focused heavy on fun and gameplay. Graphics, art, etc, can be messed up but driving mechanic is big deal. Cannot make game fun without god core mechanic.

tuning:

  • Easy to pick up, intuitive controls, balanced difficulty and learning curve. Hard to master.
  • This is accomplish with extensive tuning. Interate!
  • Do not trust yourself to tune, the game has trained you to play the game well! Bring in new people.
  • You will become jaded fast! Bring new people in for testing when that happens. Course-Meta: Get friends and family to play test but if you cannot then be aware the game is training you, fight it.
  • Me: I remeber develping flash games I would hang out on an IRC channel with other (better) devs. I’d play test a game and it would be hard, I couldn’t get past first level hard. The dev would then respond in bewilderment: “I’ve made that first level really easy!”. The game has trained them to be the first best player for their game.

Originality:

  • Novalty great way to differentiate your game from crowd.
  • Watch for pitfalls: do not redefine genre’s hard decisions. Example: do not use A & B for movement in a shooter. Trigger buttons should not control vertical view point.

How to get there:

  • Design for the player, not for yourself.
  • try something but if its not fun scrap it. Watch out: everything is boreing the first try. hard judgement cal, lead game designer’s major skill. “Is this going to work once it has been tuned?”.
  • Relentlesy prototype gameplay
  • some feature’s goodness depends on payer. Consider making something optional.
  • Learn from successful games, but not with a photocopier. Improve what they did, right and wrong.
  • Each feature must be justified: we need a menu, a function to draw a model to the screen. Critical features should be implemented fast opening time for tuning. Some needed features can be delayed, like menus. know what can be dropped in whole.
  • Big game can focus on perfecting optional things: “we can get an extra point on metacritic if we have awesome water simulation”.
  • Details/tuning can take 2-5x the implementation time.

Analyzing games:

  • Hard to be game developer without playing games. When you do you’ll notice game details.
  • Not everyone must play games: the sound mixer can get away without being a gamer.
  • If everyone else is punching the clock then at least the game design must be a gamer.
  • Mechanics:
  • What are you doing right now, why?
  • How is it line or unlike other similar games?
  • Level structure, pushing player. COD goes good job, linear but feels free because they’ve pushed you through the corridor and you feel like you could go off the path but just don’t want to.
  • How is the story advanced?
  • Sound big deal: example: movie’s deleted scenes feel spooky without ambient sounds.
  • Watching a cut scene is not gaming so devs trying ways to stuff story into game.
  • Metacritic has become major “how good is this game” measuring stick.
  • Metacritic has problems but few bad 90+ score games, and few great low metacritic scorers.
  • Course-meta: driving the car arround needs to be fun. Keep it simple since we have limited time and limited “artistic resources”, (none of us have done art).
  • Course-meta: better to make one fun course and one fun car. Not making new Grand Tarismo.

Summery:

  • Games concentrate on and are effective at different aspects of fun.
  • Know what is fun for your target audience.
  • Play games with critical eye.

At 11:32 without break for lunch yet: Starting on today’s second half, Game Architecture.

Basics:

  • Input comes in
  • time passes
  • rendering and sound go out

Elements:

  • Major subsystem, games cover most aspects of computer science.
  • main loop
  • time, huge since games are about doing things over time
  • game state
  • communications
  • entities, individual simulated actor/unit. Industry terms, also known as agents or actors.

Langauge:

  • 15 years ago only answer was C / Assembly or maybe C++
  • Now can leverage higher level languages. Brings safety and expressiveness.
  • Platform may limit choices: Xbox 360 did have C# support but that would leave you with self-porting mono for the ps3.
  • You cannot map executable code pages on consoles. Console manufacturers scared of hacks. So no JITs and most interpreted lanuages.
  • Assembly rare now: a recent Radical recent made had maybe 1000 lines of assembly
  • Some odd-balls: Jak & Dexter written in a lisp-derivative. Naughty Dog stopped using lisp, their lisp compiler guy got tired and no one else could port to ps3. Now used as in-house scripting language.

The game loop:

  • Take user and maybe network input and gerate a displayed frame + sound effects.
  • Outer loop handles setup/shutdown, menus, controls overall game flow, updates major subs-systems. Tells input to pull keys, render to render frame.
  • Styles are similar to kernels: single-threaded, multithread – cooperative (DOS, Go runtime), multithreaded – preemptive.

Example monolithic:

while (1)
{
    input();
    simulate();
    render();
    sound();
}

Cooperative:

class Task
{
    virtual void Run() = 0;
};
class Rendered : public Task
{
   void run(float time);
};

loop:
while (1)
{
   for (int i = 0; i < task_count; i++)
   {
     task->run();
    ]
}

Pre-emptive

void InputThread()
{
   while (1) input;
}

void SimulationThread()
{
   while (1) simulate();
}

Pre-emptive becomming more common because consoles are more parallel.

Course-meta: use monolithic.

Issues:

  • handle all game states: menus, in-game, paused, etc
  • Ensure order of operations: collision detection before AI state update
  • Ensure consistency of game state across sub-systems
  • Maintain modularity, minimize dependeces, reduce bugs, watch for spaghetti!

Comparison:

  • Monotlic is easy to code, but gets messy, does not scale. Course-meta: don’t worry about scaling. “if your game cannot handle 200 objects then delete 100 and ship”.
  • Monolitic gives game loop too much game state knowledge
  • Cooperative tasking system is flexible but losses clarity. Ordering is implicite. Hard to determine what is getting shared and when.
  • Pre-emptive systems are tough just like pre-empting kernels.
  • Everything is getting worse. 15 years ago only what needed to be was concurrent. Now almost every game system is concurrent.
  • No game is pure one system. Input cannot be async, just like networking or sound to lesser extent.
  • Typical AAA game looks monolithic from overview but deep down may have coop-multitask in AI for game entities.
  • Course-meta: we should be able to skip all concurerncy. Even sound which requires async can be hidden in library.

Other loop consiteradations:

  • Some event driven platofrms do not have loop
  • Example: javascript uses a timer based loop. Things like request-animation.
  • Consoles may have literal while true infinite loop.
  • Networking is in effect a subssystem on different machine.
  • Fine grained parallelism, not one thread per entity but it would be nice for maximizing core usage. Instead might use task queue.

Lunch break taken at 12:03, return expected at 13:00.

After coming back from break we had a discussion on game pricing and how Microsoft’s no-used-games policy would have pulled the power from Gamestop and allowed cheaper games. I am more sceptical about if this move would have been to the consumer’s benefit.

The discussion has turned to iPhone games and metrics driven game design. “Most game developers are gamers. Sure I’d rather have a hit game over a game which did not sell but as a gamer I find metric targeting games un-satisfying”. Dr. Drezinger thinks these casual games will develop in a different direction from regular gaming.

“Just finding any programmers who want to work on Barbie Clothing designer is a challenge”. “Its realy hard to make games for girls because you don’t have any female developers.” “The developers will ask themselves ‘do I make this tiara pink like the others? Or maybe blue?'”

Lectures start up at 13:05

Time:

  • Time is big aspect of game architecture
  • How time is handled must be considered from the begining
  • Don’t bake the time unit into your interface. Should wrap it in class.
  • Watch for time accuracy edge cases
  • Should use high resolution integer based times

Wall-clock vs Game-clock

  • Game time should be consistent
  • Game time should look like real time
  • A car running at 100km/h at 60fps should not drop to 30km/h at 20fps
  • You can accelerate time like in sports the 20 minute intervals should pass in under 20 minutes.
  • You can distort time for gameplay, bullet time or lag compensation
  • Old games often got this wrong, would be clocked to cpu clock
  • Console games clocked to whole machine, emulators must watch to not run too fast.

Subtleties of time

  • Types of clocks: real, audio, simulation, game
  • Game states might effect one clock and not another. Example: pausing only stops simulation clock.
  • Some subsystems may not accept lare time periods between running. Example: collision checking, else your cars will go through walls.
  • If a subssystem has a hard floor on FPS you might have to slow the car down or similar. So at 10fps you might run the clocks at half speed to keep the game playable.
  • Example: two incredible hulk games. Hulk had a ground-bound which moved every object around for physics sim so the game would drop to 10fps. The simulation was ran as if 30fps so everything slowed. Players thought this was a neat effect.
  • Physics is sensitive to large time deltas else things can start falling through the world.
  • Watch for sqew: Example: in Prototype the cut scenes got out of cue with the audio. In this case the game got ran off the audio clock.

Approaches to time:

Fixed step:

  • simple but slow downs slow the game.
  • Fixed has easy networking and deterministic AI. Networking wise just send people’s key presses.

Multiple simulation steps per render:

  • fixed time step per simulation
  • but run as many simulations needed to bring to real time
  • then render
  • Nice because it can handle variations in the render
  • Downsides are if the simulation is too expensive you’ll never catch up.
  • Can lead to “pop” if simluation/AI runs more often between a render.

Variable step:

while (1)
{
  new = Systemtime();
  step = sim_tim - now;
  siulate(step);
  render();
}

Works well for PC where CPU and GPU are variable power. Pacman has fixed simulate and render cost. Is non-deterministic, complex networking. Assumes last frame is usable for time step in current frame. Again physics depends on good framerate adn cannot skip. AI should work at 15fps but lower could break things.

Sub-iteration:

Like multiple steps and variable step. Run physics at fixed step and everything at variable.

Can also mix with a varaible but capped physiscs. Example: simpsons hit & run would run variable step for game and physics but it had rules for smallest physics step.  Physics was stepped twice if variable time step was too long.

Break would have been here but we’re running slow and we took the break early.

Subsystems & game state representation

Major subssytems:

  • Video games have natural decomposition
  • Course-meta: each person is going to implement multiple subsystems so keep them modular
  • Input
  • Networking
  • Rendering
  • Sound
  • Script
  • Content Loading
  • HUD/front-end
  • physics
  • Animation
  • AI

Ideally susbsystems should treat each other as black boxes. Need encapsulation. Networking is hard to keep encapsulated since it needs state from everything.

Game state

Everything in the game. Each subsystem needs game state, often overlapping but not everything.

Approaches:

  • Global state – everything has access to everything else, does not scale.
  • Global state can work for small games, “Here is the position of the orange ghost. If anyone needs it you know where to find it (global var).
  • Push / pull. Example: AI tells rendered where objects are.
  • Brokers generalize push & pull. Ferrys data between systems.
  • Broadcast / listener – tracks state changes. Message passing. Convenient. Example: Sound registers for the damage event. Is expensive and must store messages.
  • Database: needed when game state is larger than memory. Programming free saving and loading.
  • Shared Entities: each system can query a different interface on the same object. Most common method. Warning: don’t use raw pointers use IDs or handles.

Break taken at 14:07 for 8 minutes. Asked instructors about Javascript and webgl for the project.

Entities

Almost every game has an entity system.

Single-rooted inheritance hierarchy:

Naive approach.

class Entity
{
   virtual void Display(void) = 0;
   virtual void Think(void) = 0;
}

class Character : public Entity
{
   void Display(void);
   void Think(void);
}

Each subssystem can only have access to specific interfaces. Bad encapsulation since entity should not know how to display itself.

Relationships get complex and system might dictate interface. Rendering tends to take control. Example: Drawable stuff, not drawable stuff.

Multiple-inheritance can be a solution. So base class like Drawable, or AI.  Scales better, hierarchy is cleaner. Harder to share implementation/code. In C++ some performance pitfalls and is painful. More on this in lecture on C++ internals.

Composition based entities.

So an entity might have a brain which implements the AI interface.  Can share implementations by sharing components.

Entity becomes wrapper so lots of wrapper. Lots of passing by value needed since components cannot access other components.

Sorry, I was rewriting what the intstructor was saying. He was talking about an entity owning a brain and I translated that to components. He is now covered components, it’s nice to hear because I hate inheritance approaches for this problem. Long live composition!

Component-based entities:

An entity is just a set of components. Components can be removed at runtime. Needs complex system for communication. Now we can implement some components in script. So AI could be scripted for ease of development. Display component can be written in native and fast. Be cautious when fetching a full component, prefer event sending.

Entities summery:

Radical’s gamecube era were all inheritance based. Lessons learned, 20k lines in the header file for the character class. Now Redical’s games are all component. Not everything in game industry is component: Frostbite is single-rooted inheritance. So all the cars in the first need for speed based on frostbite had an invisible gun!

Communication strategies

Ways systems and entities can talk to each other:

  • Direct function calls – fast but scales poor. Long build times high coupling.
  • virtual interfaces – reduced coupling, free polymorphism, but indirect calls.
  • events – complex, should be type safe but double blind. No casts should be needed. Example of good system is QT’s signals and slots. Example of bad: WndProc, lots of casting. “heres a 32bit intergers, heres also some other 32bit integer—stuff…”. If done well brings low coupling. Warning: hard to debug. If an event is never fired then hard to find if, when, and why.
  • callbacks can assist other methods with easy bi-directionality – “do this thing, then call me back”. More direct than events, easier to debug. Nice if lambdas exist in your language. Helps async and thus helps with concurrency.

Communication summery

  • Make the AI authoritative
  • Most games never manage architectural purity
  • Prototype was done by same team as hulk but switched to components. This unlocked more experiments.

We’re an hour behind at 15:08, eating time into the scripting subject. We’re pushing scripting back and taking a 20 minute break. Coming back we’ll talk about our games and break up into groups.

We’re back at 15:23. Instructors setup a ps3 to demo some games.

We’re starting with Uncharted 2, I’m noticing the load times are pretty insane nothing like on linux with an SSD =).

While the ps3 is acting up we’re jumping into the project. Teams have been selected, I’m in team 2. We can do a racing game or karts or car combat or crazy taxi clone. Requirements are learning the pieces of the develpoment process. We must have a car AI which drives around. We’re allowed to use existing audio, physics libraries. Almost everything else should be written by use. I’ll ask about SDL.

Watch scope. Don’t plan something which would take the need for speed team four years if you only have four months.

Instructor said SDL was okay. Our game idea is a karting area death-match game.

CPSC 527 or: How I Learned to Start Worrying and Write a Virus

Just over a decade ago my university started to offer a controversial course known as “CPSC 527 Computer Viruses and Malware”. Or as I refer to it: the Virus course.

The virus course has a reputation which proceeds it. I first heard of it in second year. Word was that there existed a course held in a locked and isolated lab. No internet and no electronics allowed. Only fourth year students could enrol and an essay was required.

So fast forward to late third year when I and a friend have set our minds on this mythical course. Said friend managed to uncover the controversy dating back to 2003 over the  introduction of our viruses course.

Most of the resulting articles and press releases can still be found through a quick search. Most notable were two press releases from the Anti-Virus Manufacturers: Sophos and F-Prot. Both give an insight into why the course was not received well by all:

I just wanted to make sure that you are aware of the effects that participation in the course may have on the students’ future career. Most anti-virus companies (including ours) have a policy against hiring former virus writers for anti-virus work. What this means is that in the event that the students actually learn something useful in the course, they will most likely not be able to obtain employment in the anti-virus industry due to their participation in the course, and thus not be able to contribute to actually solving the virus problem.– Fridrik Skulason of F-Prot

Sadly it seems the university is developing courses according to what it believes will be most attractive to potential students rather than focusing on skills that will be useful to them in the security industry. One wonders if the University will be held legally and financially responsible if any of the viruses written on their course break out and infect innocent computer users. — Graham Cluley of Sophos

What struck me is the assumption both responses hold: it is Anti-Virus companies which bring relief from viruses.

I disagree. Security, and thus relief from viruses, instead derives from your ecosystem and your operating system. Which gets to my interest in the course:

  1. Have fun
  2. Becoming a better systems programmer

I have no interest in becoming a researcher for the Anti-Virus industry. Instead I want to create systems which will minimize vulnerabilities and reduce any ground a virus can win in finding a vulnerability. Meanwhile all Anti-Virus software can do is slow down your computer, take your money, and tell you it is time to reinstall Windows

Now Mr. Cluley and Mr. Skulason do raise a good question: why should you create a new virus? To answer that let us look back to 2007 during my tenure at the local McDonalds. In our flashback I am a new employee and my boss has taken time out of her day to instruct me on proper method for cleaning dishes. You see the drive through is right across from the washing station and thus it is the task of order takers to clean dishes while waiting for cars. Now I wasn’t a patient lad and protested against a practical demonstration. I asked her if she could instead just tell me how to wash dishes. To this she said:

You’ll only learn something once you’ve done it. — My boss circa 2007

Which is thus the core reason why CPSC 527 must be about writing new viruses. Similar themed courses at other universities may teach virues the way Mr. Cluley and Mr. Skulason wish them to be taught: by dissecting existing viruses. Granted Mr. Cluley is correct, this approach would better serve students going into the Anti-Virus industry. Dissecting existing viruses is the major task when writing an Anti-Virus scanner.  Thus prospective job applicants to Sophos or F-Prot would be well served by practicing their new job while in school.

Except I have no interest in the Anti-Virus industry. Instead I want to learn to think like a virus writer. I want to look over a system’s design and notice “Hey, I can abuse that!”. Not for the purpose of abuse, but to bake security into what I create.

To that end I can say CPSC 527 was a fantastic course: I wrote a virus, an exploit for a buffer overflow vulnerability, and for the final assignment a virus scanner for the other 10 group’s viruses.

All virus work was done in the virus lab: a locked lab with no internet access on machines which required a second student to vouch for your identity before you could login. The machines themselves were linux boxes running FreeBSD in a locked down virtual machine. Even if a malicious student did brings the virsues out of the lab there would be no targets to infect!

The virsues we wrote were not complex or modern. My current value to the Russian Mafia is still about zero dollars.

With that said my virus did win the prize for best virus, which in this course means it was the worst and most annoying virus.

What my virus did was launch a rootkit into kernel space. This rootkit would then sit and count how many files a thread opened. Once the thread has opened N files it got marked as an anti-virus. Then instead of killing the undesirable scanner, we would deny all write requests. Thus a scanner would work for the files in /bin but any warnings of virsues in /sbin or /usr/bin would be silenced. The net effect was to make programmers think their scanner was broken!

The rootkit was so effective even I got caught by it! My scanner would find the infected files in /bin but not the ones I knew were in /usr/bin.

Beyond pride, writing the rootkit was enjoyable, fulfilling that goal. Spying on the scanners required hooking BSD’s system calls and stuffing data into unused fields of the thread’s task struct. I even booby trapped the system calls table: any attempt at unloading my rootkit would leave non-NULL yet invalid pointers in the system call table. Thus any calls to open or write to files would crash the kernel.

Towards the intangible goal of improving my systems programming I am now quite paranoid. My programs now enjoy user input escaping far beyond anything reasonable. I found myself escaping even the input I got from the kernel! The kernel being the program which runs my program and already controls the entire computer.

As for systems design, my fervour for mobile and Firefox OS has only increased. We need to replace the Win32 ecosystem with an ecosystem that can be sandboxed and we need to do it without throwing user freedom to the dogs. To that end the web is perfect. Web browsers have undergone years of security stress testing in the most hostile environments imaginable. The web represents an ecosystem developed from the begining to be sandboxed, it is not even possible to self modify binaries!

In general we need more programmers to take courses like 527. University students no longer write virsues for fun or to prove a point. Instead we have rogue governments attacking their own people. The risk in teaching students virus writing is now far outweighed by the benefits. We need programmers who will make security a priority. At Microsoft I learned that “UAC is not a security boundry“. What this means is that any vulnerability allowing bypassing of UAC will not get patched. They’ve resorted to this useless stance because there are already many unpatched vulnerabilities. The net effect is users are running under admin accounts which feel like regular non-all-powerful accounts.

So if you have a chance: please enrol in CPSC 527, or a similar course if your university offers one. If enough people take courses like these maybe the next UAC will not happen.

What is UCOSP?

UCSOP stands for [U]ndergrad [C]apstone [O]pen [S]ource [P]roject. It is a good name as far names go but the details are what make UCOSP unique. UCOSP is a course ran twice per year at several Canadian universities. About 80 students participate per year and said students are supposed to be some of Canada’s best, but they let me in any.

For the kick off of the semester, the students meet together once as a team for a hackathon. Our hackathon was hosted by Mozilla in their Toronto office. All flights were even paid for by Google as part of the Google Open Source Programs Office. This is the same office which hosts the Google Summer of Code program.

Like a regular capstone project the course is results focused and hands on. Also like a regular capstone course you’ll have a professor watching your progress. The difference is in your team. UCOSP’s goal is to introduce students to distributed teams. In your classes you’ll have worked on centralized teams. My university classes had tons of these forced group work. Yet, none of my prior classes introduced distributed cooperation.

Why are Distributed Teams special?

A distributed team lets people live where they want. Instead of trying to move the whole world to Silicon Valley a distributed team may have members on every continent. Many of the world’s best engineers like living where they live, and they’ll sooner work for an organization which does not force them to move. Of course for others moving is not a problem, but if someone earlier in life has decided to move there is a good chance the reasons for that move have not changed. Nor may they change just because you want to hire them.

A distributed team also brings other more practical benefits. By staying out of the insane property market which is the Silicon Valley area, you’re increasing employee’s take home pay by tens of thousands of dollars per year, for free! And by cutting out the commute time, team members will be saving an hour or more per day.

When you're distributed you need to be more explicit in what you say. For example, this poster is not sugar coating the message.

When you’re distributed you need to be more explicit in what you say. For example, this poster is not sugar coating the message.

Challenges of Distributed Teams

Distributed teams bring their own challenges which may work against student’s weakest attributes: being self-driven and getting things done. Without supervisors or near due dates, many students find it a challenge to get work done. This is a problem for distributed work, you’ll have no one watching over your shoulder. You must know what a reasonable commitment is and have the will power to see the work to completion.

Communication also takes on a different form for distributed teams. For what it is worth: in person conversations are more effective than you might expect. A quick chat with a group member before class could get you unstuck on a nasty bug. Likewise centralized teams have an easy time of sharing progress, just talk about it over lunch. Instead distributed teams must leverage email or wikis. Documenting the project, the meta project, goal, reasoning, all take on extra weight. As does communication finesse. Online we lose non-verbal clues so you must watch how you say things and yet be more direct in saying them. Leaving details as implicit will generate confusion since there is little context for readers to read in between the lines.

In general, I think these skills come from practice, which is why UCOSP is special. The regular student might never contribute to a distributed project, open source or not, while in university. UCOSP is the rare course which gives students this experience while earning course credits.

If you’re a Canadian Computer Science student you should head over to ucosp.ca and see if your university offers UCOSP, or a similar course.

My semester in UCOSP

If you do not know what UCOSP is then you should consider reading my explanation post: What is UCOSP? Otherwise in short: UCOSP is a practical experience for Canadian Computer Science students to work on open source as a distributed team. For my UCOSP project I wrote a zip extraction library for BB10.

How I got enrolled

I found out about UCOSP in mid June while I was still at Microsoft for my internship. My self-assigned task that day was to arrange all my courses for the next school year. Many of the courses I wanted to take required pre-approval from professors which in turn required an essay, so planning ahead was vital.

While finishing up one of my essays I saw one of my friends online so I chatted with him. I tried to convince him to take one of the cool courses along with me. Meanwhile he asked if I had been invited to some course called “UCOSP”. In his words: “open source project course”. Little did he know that those words describe the perfect course ever! Imagine getting credits for doing open source projects!

Yet I had a stumbling block. Enrollment required an essay due the next day. Meanwhile said friend has one of the highest GPAs in our year and thus got invited. My own GPA is not bad per-say but it must not have been high enough for the invitations. Which makes sense since UCOSP is a new program at our university, they would want to run the pilot with our best students. Yet this means my work is just that much harder: I have to write an essay that qualifies for UCOSP but also one that convinces the professor to risk letting me in on his shiny new pilot course.

As if that was not hard enough my friend wasn’t sure he was allowed to give me the name of said university professor! So I sent an email off to the UCOSP steering commity asking for the contact. They replied the next morning with the professors name, and I sent off an essay email soon after. Five days later I got an email saying I was accepted.

Our Hackathon

At the beginning of every UCOSP semester all the students along with their project mentors are brought together for a hackathon. This semester the hackathon was in Toronto and hosted by Mozilla. Next semester I hear it will be in California hosted by facebook!

IMG_20130919_105120

Calgary’s Glenmore Reservoir from the airplane

Being on the wrong side of North America meant I had a long flight on the 19th of September. To make the trip more nerve-racking I’m far too cheap for a taxi. Instead, after my four hour flight I took the Airport express bus to the subway, after which I walked to the University of Toronto Campus for the evening meetup.

IMG_20130919_165118

View from the Univeristy of Toronto’s Computer Science’s upper meeting room.

While the hackathon wasn’t starting until the next day, we had an optional meet-and-greet at the University of Toronto. Somehow everyone from my team made it except our mentor. Which is okay since there were only one or two mentors there in total. Beyond some ice-breakers, we also met some of the professors organizing UCOSP. In all it was a nice low key event.

 

IMG_20130920_065936

Left half of Mozilla’s entrance way. I should have taken a fuller photo but I only even took this one for the banner’s sake.

The walk from our hotel to Mozilla was long and took us through Toronto’s banking district known through out Canada as Bay Street. We ate breakfast at First Canadian Place, I had eggs.

IMG_00000010

The hackathon in full swing! Look at that coding! The intensity! You can tell work is getting done because no one is talking.

At this point I have to apologise, I took almost no photos of the hackathon itself. I have a few but most are test photos from my developer phone. Speaking of which, our mentor gave us each a developer phone!

IMG_20130920_070913

Our BB10 developer phone!

I have far too many smartphones at this point but this is still the first phone someone has ever given me. The phone itself is a Blackberry Q10 with a different shell. Since matte black is my favourite color I have to admit that I prefer this dev phone over the Q10. Since the BB10 emulator requires VMware under linux, which I do not own, this dev phone was critical.

You might be curious at this, I have yet to tell you what we were even working on!

Our Project

My team’s project was to port popular Cordova plugins to BB10’s new Cordova based html5 platofrm. Cordova is a compatibility layer over a mobile platform’s app embedded web browser which exposes phone features as web APIs. In iOS terms is wraps the WebView and provides access to memory cards, cameras, and etc. Cordova is the offical open source name of PhoneGap after Adobe bought and open sourced PhoneGap. At the time of our hackathon, Blackberry’s offical html5 platform was proprietary and called WebWorks. Our work was to coincide with the development of WebWorks 2.0 which was based on Cordova. Plugins are extra APIs. For example the API I ported exposes extraction of zip files. Our mentor set a goal of four plugins, of course more would be welcomed.

Back to the Hackathon

IMG_00000018_hdr

I might have been playing with the phone’s built in instagram-like filters when I took this photo, or maybe the HDR. Otherwise I cannot explain the lighting.

For our part this hackathon. was spent setting up our development environments. Sad to say but the setup process is a PITA. Clocking in at 15 steps when I wrote it up, there is zero chance I would have figured it out without our mentor’s inside knowledge of BB10. At this point I should mention that our mentor works at Blackberry so in a way our project was an extension of BB10’s development.

In a far too real way, half our pain was rooted in BB10’s closed nature. We had to request keys from Blackberry and these keys were only batch generated every two hours. Once the server sent you a key, you had to setup two different processes for package signing. The dev phone themselves needed to be put into dev mode after every reboot or you would hit misleading error messages. By far, the most painful mobile dev environment I’ve ever setup and it stands it stark contrast to Firefox OS’s dead simple html5 environment.

IMG_20130921_155015

This may have been the morning of the 21st, or the evening. In either case we were all tired.

 

IMG_20130920_102218

Midway through the 20th, Mozilla gave a demo of asm.js. I’ve seen it before, but it was exciting to see a demo of the tech from Mozilla themselves.

IMG_20130922_090127

I’ve never been on a real tram before so it was nice to have the chance on my way back to the airport.

In prior years UCOSP tried to go right up until the offical end hour but this year they organized things to allow early departures. This gave me plenty of time for my airport commute.

IMG_20130922_171439

Somewhere over the prairies. Judging by the abundance of lakes, I’d guess Manitoba.

Rest of the Semester

Once home I met with my professor on Monday to plan how the work would go. He suggested writing a list of commitments for the week in our weekly monday meeting. Then during the next meeting I had something concrete to report on. For the first week I took it slow to catch up on missed school work. In the later weeks the goals were sub-milestones for my zip plugin.

At the semester’s end he had us UCOSP students give a presentation on our projects to each other and to the students in his other capstone course.

In the end we hit our goal at four plugins but there is a chance we were still the least successful this semester. We got the work done and I’m proud of that, but we didn’t act as a distributed team. We set out as two subgroups of two devs a group. Due to some hardware, failure my partner didn’t have a development environment setup until half-way through the semester. By this time I was deep into my zip extraction plugin so instead of joining me on the zip plugin, my partner decided to start out on his own plugin and salvage what was left of his semester. I regret not keeping in better contact with him. I had pinged him a few times but I held back checking too often for fear of being nosey.

Thus, I cannot report doing the fun distributed team things like code reviews or hanging out on irc. With that said, please don’t let my experience discourage you. Stuff happens and sometimes things go wrong, but in general they go right. The other UCOSP students from my university reported lots of code reviews, wikis, and even team blogs! Just imagine, next year you could be hanging out with team members from across Canada working on your own UCOSP project! Apply today!

Ubuntu 13.10 Blackberry OS 10 Cordova plugin development setup

After upgrading to Ubuntu 13.10 the QDE setup for 13.04 needs new dependencies. The change itself is small, just install the following libraries before you run qde:

sudo apt-get install libgtk2.0-0:i386 libpangox-1.0-0:i386 libpangoxft-1.0-0:i386 libidn11:i386 gstreamer0.10-pulseaudio:i386 gstreamer0.10-plugins-base:i386 gstreamer0.10-plugins-good:i386 gstreamer0.10-ffmpeg:i386libcanberra-gtk-module:i386 libcanberra-gtk0:i386 libcanberra-gtk3-0:i386 libcanberra-gtk3-module:i386 libxtst6:i386

If you do not install those you will get errors like:

After upgrading to Ubuntu 13.10 the QDE setup for 13.04 needs new dependencies. The change itself is small, just install the following libraries before you run qde:

If you do not install those you will get errors like:

java.lang.UnsatisfiedLinkError: Could not load SWT library. Reasons: 
	/home/danieru/.bbndk/configuration/org.eclipse.osgi/bundles/364/1/.cp/libswt-pi-gtk-4236.so: libgtk-x11-2.0.so.0: 共有オブジェクトファイルを開けません: そのようなファイルやディレクトリはありません
	no swt-pi-gtk in java.library.path