Requirements for our code

Completeness, correctness, unambiguousness, reliability, consistency, and traceability.

Any module should ideally fit in a single page.

If you're copying and pasting code, it's wrong.

How well do we understand system boundaries?

Aesthetics.

Reusable components in our code.

literate programming and the rational unified process

The wikipedia article on literate programming is very well laid out and was a good read. There are a lot of literate programming "languages" out there! I wonder if there is any style that generates test cases as well as code. It may be good and all to interweave thoughts and comments with the code in a natural-language type flow, but I think it would be hard to debug this, given the amount of text you have to search through, and all of the chunks of the same piece of code being scattered about. Maybe you compile it into source code, and then debug it, and then figure out the pieces you debugged and inject them into the original? Sounds a little too hard...

I would like literate programming if we designed a language that could take natural language and produce code, resolving ambiguities in the natural language by waiting until the last moment to evaluate and dynamically type any portion of the program, interpreting the language by the surrounding state at the time of execution. If it still can't be resolved, throw an error and let the user revise their language, or be more specific, or whatever.

Of what is out there right now, I think noweb would be good to learn.

On to RUP:

Developed by IBM, this process is very detailed, and seems interesting to learn. A summary of it is very interesting (about 20 pages). It turns out that it is all about using UML as a way to communicate design, requirements and architectures across teams. A bunch of tools are employed to keep the models up to date and to support bookkeeping.

I definitely see how important an easy tool for bookkeeping would be!

And the IBM 6 best practices for a software development team are:

1) Develop software iteratively -- agile methods I think work just as well here
2) Manage requirements -- they emphasize use cases and scenarios, as in agile!
3) Use component-based architectures -- I think Python is a great example of this in practice; developers of Python have made dozens upon dozens of robust and useful components for the language
4) Visually model software -- this is where UML comes into play. I think we need to use this more.
5) Verify software quality -- Here I think agile is better - test first!
6) Control changes to software -- this is what good version control software is all about, right?

Here's a visual of how they envision the "process":




















I think our team could do better with a vision document and a bigger overall picture of what we want to create.

The rest of the doc goes into details about this process.

class monday

I've been watching the video from Monday's class. My team has done a great job responding to what Joe brought up, based on our conversations in our meeting today.

I am very impressed by what everyone has accomplished, and it makes me committed to work harder this next week to output as much as everyone else is.

long, empty week

Somehow I accomplished nothing over spring break.

Well, I got my job back. Took a drug test. Read a computer science book. Accepted a counteroffer from a bank. Wrote a script for the Independent Sets problem. Wrote some variations on a self-assembler and missile in the Linden language. Spent some time on the next kaggle competition.

What has been delaying me here? Poor time management, I guess. Not focusing on this as my priority. I catch up on what my team is doing, and then I run out of energy. Often not having access to the internet -- I fixed that problem yesterday. I will renew my focus tomorrow.

Asana collaboration tool

http://asana.com/2011/02/asana-demo-vision-talk/

this talk is about a cool new tool. Their goal is to have a product that is as fast as notepad, that updates immediately across the network, that holds all "truth" about a project with multiple ways of distilling and viewing the data, and that keeps all conversations about the project attached to the actual tasks in the project.

I like how it is very notepad-like in updating and changing things. Every task has a chat attached to it, where people can "follow" the task and be emailed when a note about it updates. Responding to the email also updates the task chat, like in google groups. It is easy to move assignments around to people. When a task is finished, all the followers are notified.

Very social-appified, which makes sense because they came from Google then Facebook.

This tool could really help us with our team meetings and coordination, much better than Google sites. It looks like they also built in ways to collaborate your software and components with other developers, so you can use your social networking to get tools, information and help you need to progress as a company. Their goal is to have "business to business virality".

They also developed an in-house web app programming language to speed the design and deployment of that kind of software, called "lunascript".

I applied to see if our team can beta-test their product.

high-level stuff we need figured out

-- Design patterns

-- Coding standards

-- Commit patterns and how to stick to our process

Evaluation of Ekaterina Davydenko's proposal

Dynamic Traffic Control System

Total evaluation criteria score: 25 / 40

Actually, not bad compared to a lot of proposals. It only misses what many of the other ones also miss.

Part I

As far as I can understand it, the Dynamic Traffic Control System idea consists of building large overhanging and side-of-the-road electronic signs that can be either manually controlled or automatically adjust to road conditions. It wasn't clear in the proposal how much of the system would require continual updating from human personnel, and how much would be able to run on its own.

The system would also be connected to all the traffic lights in a city, have a centralized database (what for, I don't know), and be manipulable by law enforcement, emergency medical vehicles, DOT people, and firefighters.

Part II

One thing I don't get is why this system would be centralized. A decentralized, robust system seems to make a lot more sense to me. Rather than a centralized database, distributed servers that control their own portion of the roads and communicate with each other and whoever logs into them would make more sense.

Syntax score: 5/5

The structure of the proposal is professional, and the spelling and grammar are accurate and appropriate. The writing is, for the most part, clear and sufficiently detailed. It could cover more about how personnel will be trained to use the system and the interfaces they would have to learn and use. The story is consistent and well-structured.

Plausibility: 2/5

The project cannot be completed in 12 weeks. As it stands, full implementation, testing with agent-based models, and deployment (along with the training of relevant personnel) would take a long time. Furthermore, a centralized system for managing traffic conditions would be vulnerable to exploitation and reliability issues. Also, what happens when different bureaus decide to give opposing directions for the same chunk of road? Whose decision becomes final? Would a city really consent to such a project? It will need a lot more explanation and convincing data on other such systems.

If a decentralized system were chosen instead, how would it do better than other systems already in place, in terms of manageability, cost, and projected savings in time and money for consumers, and lives for medical personnel (and when it comes to traffic accidents, etc).

Support: 2/5

The arguments for the need for such a system are sound, but I find no convincing evidence that yet another costly city road project like this system would really benefit the city. For instance, traffic lights are very complicated to perfect, and the systems already in place have been tested and tweaked to be about as good as they can get. They also are remotely configurable and change with the level of traffic. The new system would likely not do anything to make them any better.

Novelty: 2/5

The idea of managing traffic conditions is not new. The approach offered really doesn't sound worthwhile. Just doing a quick search brought up, for instance, http://www.cttraffic.com/pages/CISV.html, which details the complex devices already employed in traffic control. They use fiber optics, video relays, remote monitors, exclusive relay lines and control software already. All this is very expensive and requires approval and control from various state authorities.

There is a book that covers many details of traffic control that is published by the Department of Transportation: http://ops.fhwa.dot.gov/publications/fhwahop06006/index.htm. It covers controllers, sensors, managing urban and suburban roads, freeways, and integrated systems. It covers communication and information systems, and design, implementation, and management of Traffic Control Systems.

For a history of traffic light management, see http://ops.fhwa.dot.gov/publications/fhwahop06006/chapter_1.htm#1-5 and sections five and six. I also think it would be useful to review Figure 1-3 from that page and try to see how the proposed system would fit in it. Table 1-2 from that page really gives a sense of the broad scope of details that need to be covered within traffic control. It gives the sense that the proposal largely underestimates the scope and difficulty of managing traffic, and overestimates the impact that installing more traffic signs around a city or freeway will have. The ability to post warnings about delays, etc, also already exists in many big cities.

I found some interesting research on decentralized traffic control: http://www.physorg.com/news114355988.html. It actually doesn't seem that useful when you really look at it. Not without even more research and simulation.

Stakeholder Identification: 4/5

"Private companies" are mentioned, but it sounds like the whole idea is intended for the DOT. This is a project that is meant, it sounds like, to be funded by tax payers.

Scope: 5/5

It's pretty clear what was intended. I liked the use of a diagram to show how components were to interconnect.

Profit/Impact: 4/5

The amount to be made from the DOT or other groups was never stated, but the impact on drivers and law enforcement, etc, was made quite clear. If this was to be a privately funded company, selling their services to state and federal governments, there would be a greater need for projected profits. As it stands, the true costs of all the hardware and construction for implementing such a system were never really covered, either. This would, therefore, make a great government project =).

Security/Risk: 1/5

The project proposal didn't really define any risks beyond that a "major concern will be the dependability and ... integration ... requirements of the system".

Some of the major risks I could think of off-hand were as follows:

1) There is danger of traffic warnings being ignored / no mention of social engineering to get the job done. Thus, a lane may be attempted to be opened up through signs flashing to keep it free for emergency vehicles. During rush hour on a busy freeway, without physical barriers, how many people would disregard such a rule? Enough to slow down cops anyway? The same applies for medical personnel. In the event that no one sees any immediate reason to follow a mandate, many likely will break the mandate. It also encourages disaffection with the system, when it tells people to do something but doesn't offer immediate positive results.

2) The securing of the systems is never presented, nor the ability to deal with conflicting orders from different organizations. The way to handle events like server failure isn't mentioned, either.

3) Sensing systems and their management are not mentioned. "Database" doesn't say enough, if that was meant to incorporate data from video feeds, scanners, and human input.

Self-Evaluation

some guides:

http://www.wikihow.com/Write-a-Self-Evaluation

http://www.evergreen.edu/washcenter/resources/acl/e3.html

my self eval:

This is a 3 credit hour class, and I have been treating it like a 6 credit hour class. I believe I have been going above and beyond what is expected of me by the professor. Part of my doing this has been because of my missing the first three weeks of class. The other part has been because I continually strive to perform at the top of my abilities. Sometime, I strive too hard, but I feel the need to do what it takes not only to get an A or A+, but also to learn the most that I can to prepare me for future things.

Examples of efforts I have made include catching up on all the class assignments from the first three weeks. I reviewed one proposal completely, one partially, and I read all 29 proposals thoroughly, giving a brief review and rating. I managed to turn in my proposal in time to give a 2 minute presentation. I also had my proposal reviewed, and edited my proposal in response.

I also have actively maintained a blog, researching and taking my time to output valuable comments that I can later use. Of the 4 1/2 weeks I have actually been in this class, I have given over 20 blog posts, 17 of which are very lengthy, researched, and detailed. I have thus exceeded expectations for blog posts per week that I've had this class, and have slowly but surely been catching up on the 9 posts for the weeks I missed. I have eliminated fluff from most of my posts, thus not always blogging on the topic the teacher suggested if my research in that area returned sparse results, and I had a replacement topic for the day.

I have hustled to learn techniques for success in programming on a team and to think of a viable project for our time frame and that would actually have economic benefits. I also learned voting techniques and Roberts' Rules. I've learned new web programming and interaction techniques, and shared them actively with my team. I have gone the extra mile in working with people from the school of management, and learned how to start a company and seek funding for our team. I am working on getting us connected with several different venture capitalists, including the Angels. I will be trying to see if a representative will come and speak to our class.

I've expected to learn from this class, in broad sense, everything that it takes to run my own startup successfully. I figure the best way to learn is to actually do it, and this class has been the perfect springboard. My expectations have included the objectives from the class webpage: to "gain experience in project development begin[n]ing with concept proposal, specification, implementation, and maint[e]nance in a team setting".

I have learned a tremendous amount each week since I've been in the class, both through my own research, and through participation in class and on my team. I have sought to share what I learn with others so we can all learn together.

As a leader, I have sought not only to be productive and make my team productive, but I have sought to balance the leadership roles and give everyone the chance to learn and grow. I try to follow Dale Carnegie's principles in How To Win Friends and Influence People, though I've stumbled a bit there. I do review my behavior and seek to change for the better.

My greatest weaknesses include my tendency to take too much work upon myself, and get burned-out thereby, not relying enough on my team members. They also include my tendency to stray from established schedules, to not stay on task, and to not follow up with others on my team or ask for help enough. I also am a huge procrastinator, though not always on purpose. It's more of a time-management thing, where I budget too little time for some tasks, and take too long in others, and then end up rushing and behind in a lot of things. I do evaluate myself on these things and seek to improve.

Self-ranking based on above discussion (with somewhat arbitrarily chosen metrics):

Participation (in class and in team): 5 / 5

Learning: 5 / 5

Self-direction (initiative, motivation): 5 / 5

Follow-through with responsibilities: 3 / 5

Completion of Assignments: 4 / 5

Progress towards course objectives: 5 / 5

Thus, my total self-evaluated ranking, giving the above criteria equal weight, would be about a 27/30. This equates to a 90%, which based on what I've observed in our class, should put me in the top 5 students, with a well-deserved A.

points in my favor:

- I created the Farmville bot idea

- I fought hard to get my idea to the top five and keep it there

- I researched Scrum and taught it to my team

- I have dedicated many hours to solving the "hard" details to get our project to work

- I have put our team in touch with the Anderson School of Management, and through our contact, the Angels investment group and other investors

- I have maintained a focus on what it will take to actually build a successful company from scratch, and am committed to seeing our idea through to the finish

- I maintain focused on making our entire team focused, motivated, and efficient at what we are undertaking

- Our team has been among the most "hit-the-ground-running" types

- We focus every week on improving ourselves and improving our process and communication

- I have been learning a ton about managing a successful team, and also I have been learning a ton about internet protocols and web applications

class goals:

- we have the goal to learn what it takes to be successful in the "real world"

- our goal is to learn to handle all the facets of a real software development cycle

- we have to learn how to work efficiently on a team and with a strict time budget

my weaknesses:

- taking too much work on myself rather than relying on teammates

- not maintaining my blog with everything going on in class, preferring instead to only write down what I think I personally need to remember or keep

- I don't follow up with people often enough, and don't follow a schedule well either

startup success rates

Here is a good, recent article about startups.

There is actually a 60% success rate by their numbers, and they suggest several factors for success:

2-3 person teams are better than large teams. I believe Jim Collins is one of the best sources for how to be successful also. A large part of success has to do with the team and leadership. A visionary leader is more important for a startup than a "committee". Being a young team gives us an edge. We need to be fast and decisive.

The success rate seems WAY inflated to me. Here they mention that the success rate is closer to 18% for first-timers, and 20% for people who have failed before and are trying again. People who have been successful typically see a 30% chance for success on their next venture. These numbers feel a lot more accurate -- and their data is supported by this paper, which also mentions that being funded by experienced venture capitalists gives you higher chances of success - and more initial funding.

Lastly and probably the best of all this besides the Jim Collins books, is this Paul Graham article on startups, and why you should do them. One thing he points out that I think is important is to make sure to have a cofounder in what you do. Also, most of the time the idea the startup was founded for changes before the product is finished. Another great reason for Agile teams!

online service horrors... and potential

So hosting complicated programs online causes much pain.

This discussion talks about Zynga adding up to 1000 servers (including the Amazon cloud servers) a week to keep up with their huge host of customers. What with over 200 million customers per month, they needing a server for every couple hundred people, they now have tens of thousands of servers running their games. Running through a petabyte of throughput a day. Insane. 7% of their users, on average, are online playing at any time. Even cutting that number in half so it's 7% of the monthly users online at any time during 12 hours of every day, that means that over 5 billion hours get spent on these games every month.

What could we accomplish with 5 billion hours of free time? Hmm...

This is a game company site outlining what they themselves did, combining Amazon's cloud with their own servers, running Ruby on Rails and Javascript combined with Flash and their own custom "GAML" engine to get things going.

The site offered a lot of other great details. For instance, through micropayments, a well-performing online game world company can expect to make between $1-$2 (amortized) per user per month (ARPU). From here you can see that estimates are that Zynga makes the most from their Poker game at $2 ARPU, whereas farmville only pulls in $0.50 ARPU. Their war games, surprisingly, are pulling in about $1.50 ARPU per month, but from the first link's discussion, it probably costs them a ton to move all the data around in those ones.

The original discussion from hacker news also mentions that the average server should be able to handle about 500 requests per second on average and 4000 per second during peak times. These are smaller requests, like what we'd be doing with a bot that is verifying itself. If our bots check in every 15 minutes, that means we could easily manage a hundred thousand, or even a couple hundred thousand bots with a single server.

This is a cool article about how Zynga uses Amazon's EC2, with an Apache PHP front end, memcached system for active user play, and mySQL for their databases. Here is a blurb about memcached and how the Black Hat community pointed out a fatal error many companies were making, exposing all that data to anyone who cared to connect and listen. Zynga probably has it patched by now. Here is where you can find an article and tools for messing around with unsecured memcaches.

notes from class

critical path = the stuff that needs to be done, and in order, for project to complete

self evals due before next Friday

Long day

So today I had a meeting with our business contact. He talked about forming an S corporation rather than an LLC. With that kind of set up, we would be taxed like a business partnership, but if anything happened the company would be liable for problems rather than us programmers. Also, we could set up hedge funds and authorized shares for our company.

We also discussed funding that is available to us, so that we programmers could get paid and receive funding for our needs over the summer -- housing, travel, phones, food, family expenses, and bonus incentives among the categories we would get funding for.

We also discussed other projects "in the works" that I am not privileged to share.

We also had our second pre-sprint meeting, which was good.

We are modifying our format, which makes us not entirely scrum-tastic. For instance, we are sprinting in 1-week intervals. That is more eXtreme Programming-like. However, unlike XP, we are not changing our sprint todo list until the next sprint, and we also don't necessarily tackle highest priority items on the product backlog.

This article is pretty insightful on the contrast between scrum and XP. They make some good points. I definitely think that the scrum was a good way to go for our team. We are making progress and learning how to be more efficient as we go.

I also like this. It says that the best project for agile methodologies are ones where there is urgency in the work, and the tasks are new and complex for the team. I think our project exactly fits that description.

The highlights of scrum include having self-organizing teams, and a strong focus on the product that the user sees.