Data is ones and zeroes | Software is ones and zeroes and hard work.
My grad student Mike Conley decided several months ago to look at code review for his master’s thesis. After exploring a few ideas (including badges and achievements for reviews), he has put together a plan that might tie several things together quite neatly. He and I would both appreciate your feedback.
The final exam for my CSC302 software engineering course was due this week, and I thought I had come up with a good–and fair—question to put on it. Four out of four of the other instructors I spoke to, however, didn’t like it, so I left it out. I’d be very interested in hearing from present and former students (and other teachers) what they think.
This term, I organized the CSC302 students into six teams of about ten people each, and put each team to work on an open source project. Most of the marks in the course are given to the team as a whole, so I wanted a way to assess individual contributions. Having students evaluate their peers would be easy, and in theory would reward people who’ve done more, and penalize people who’ve done less. However, peer evaluation is notoriously vulnerable to gaming and social pressure.
My thought was therefore to measure how well people understood their own contributions, rather than the actual contributions. I thought I could have everyone rate both themselves and all of their teammates. An individual’s grade would then be based on how closely their self-score matched the scores given to them by their teammates. If your teammates all gave you 2/10, for example, and you also gave yourself 2/10, you’d get a high mark for knowing that you’d done a poor job.
Well, you can probably see the flaws, and having thought about my colleagues’ comments, so can I. For one thing, it seems unfair to give people a good grade for doing a poor job. It also seems unfair to grade them twice on the same work (once during term for doing it, and once again at the end for having done it). And it would still be susceptible to gaming, though surprising students with it a final exam would reduce that risk.
All of which raises an interesting question. Peer evaluations are routinely used in industry as part of annual performance reviews; bonuses and promotions routinely depend on what your colleagues think of you. If it’s OK to use them there, why is it so difficult in a course? If anything, there is more incentive to game things in a company where you’re likely to work with people for years than in a course where you’re only working with someone in one or two courses.
And there’s food!
I wanted to go to SIGCSE (the computer science education conference) this year, but breaking my self-imposed no-fly rule once made me feel guilty enough. Lots of people have been blogging about it, so I have lots of catching up to do (and lots of videos to watch). The theme for SIGCSE 2011 
, which is being held in Dallas, Texas, is “Reaching Out.” The symposium will focus on connecting with colleagues in other fields and developing interdisciplinary courses and research that integrate computer science and other fields. If all goes well, I hope to have something to say by then about Software Carpentry.
I just (finally) watched the demo video for Andrew Bragdon’s CodeBubbles. You’ve probably already seen it, but if you haven’t, check it out: it rocks. Like Kael Rowan’s Code Canvas (a Microsoft research project), it imagines an IDE that is more than just a bunch of 1970s-era TTYs in a frame. I think of these as bottom-up efforts: both still accept that source code must be ASCII tokens, and do the best they can from there. In contrast, Intentional Software’s still-in-beta product (described in this talk) goes further toward treating source “code” as a model in the model-view-controller sense, so that rendering and interaction can be comprehensively customized. I still believe that sooner or later, the maker of a proprietary language (most likely Wolfram Research or The MathWorks, but Microsoft is still in the running) will proudly announce a breakthrough in this area, and that everyone else will scramble to catch up, while graybeards on the sidelines point to the original Smalltalk of the late 1970s and grumble that it has all been done before.
Here’s an interesting complement to both the GSoC model (students work full-time for 12 weeks on a project) and the UCOSP model (students work on a project as one course among N during term): hire a bunch of interns from MIT during their one-month inter-term break, cram them into a room, and give them each one independent task. I expect outcomes will be very good, as it combines collocation (which GSoC doesn’t provide) with a chance to focus on just one thing (which UCOSP doesn’t offer). It also helps that they’re recruiting from a pool of students with relevant experience, and who have already internalized most of the required working practices. Wish we could try this model here, but (a) we don’t have an inter-term break, and (b) we don’t have money.
If I want to request a book at the University of Toronto Library, all I have to do is log in to their web site, look up the book, and click “reserve”. Unless the book is in someone’s study carrel. In that case, I have to go to the library in person and fill in a recall slip. Can’t do it through the web or over the phone, even though the information about the book being in a carrel is in the same computer as everything else. According to the library I spoke to, “The computer will not produce a list of books to be retrieved from carrels, closed stacks, or basement. I believe people in [the library] are trying to automate the process but at present they have to use this process.” This isn’t stupidity or sloth: it’s just too low a priority to have been done yet.
The “Y2Gay” post is similar: millions of printed forms, and thousands of databases, assume that marriage involves one male and one female, not two of either. (I have this picture in my head of government IT staff sitting down to a Monday morning planning meeting back in 2006 to hear their boss say, “OK, I know this wasn’t in plan, but…”)
So, the next time you’re wondering why open government/open data is taking so long—why “they” don’t “just” make everything public—spare a thought for the poor, overworked database admin in the basement who has to turn good intentions into SQL.
No, really, that’s what MyTTC.ca does to find transit routes from A to B.
Brief article from The Economist about how some technological innovations help empower women, while others do harm. Not much depth, but worth reading. I wonder where stalker-assistive technologies like geotagging will eventually be found to fall?
Interesting post about the future of book publishing: it’s brighter than you think, but quite different from today. And yes, I remember Sam’s too…