Assignments
Data-Intensive Distributed Computing (Winter 2024)
CS 651 Final Project
The final project is a requirement only for graduate students taking CS 651.
The topic of the final project can be on anything you wish in the space of big data. Anything reasonably related to topics that we covered in the course is within scope. For reference, there are three types of projects you might consider:
- Implement a big data algorithm in MapReduce or Spark: choose a particular big data algorithm (for processing text, graphs, relational data, etc.) and implement it. Ideally, the implementation does not already exist in a library or open-source package. Since we want you to implement the algorithm from scratch, it might perhaps be too tempting to simply copy existing code—see notes on academic integrity.
- Learn and explore a (new) big data processing framework: although we discussed a variety of processing frameworks in class, the assignments focused on MapReduce and Spark. Here's your chance to learn a new processing framework, e.g., Spark Streaming, GraphX, Giraph, Flink, etc. The project would involve learning to use the processing framework and doing something interesting with it. The "something interesting" might be a data mining algorithm, although note that the expectations would be lower than building something in MapReduce or Spark, since learning the new framework would form an essential component of the project.
- Perform some interesting data science. Is there a particular dataset you'd like to explore or analyze? Your project could involve performing interesting analytics on a dataset—here, the focus would be the analytical product and the insights gleaned, as opposed to the raw algorithms themselves. However, a superficial analysis with existing machine-learning libraries is not enough.
You may work in groups of up to three, or you can also work by yourself if you wish. The amount of effort devoted to the project should be proportional to the number of people in the team. As a guideline, the level of effort should be comparable to two assignments per person.
When you are ready, send me an email describing what you'd like to work on. I will provide you with feedback on appropriateness and scope of your proposed project. The "soft" deadline for this proposal is Nov 11, 2024. There is no penalty if you miss this deadline, but it is in your best interest to not leave this proposal to the last minute.
The deliverable for the final project is a report. Use the ACM Templates, or something similar. That is, there is no specific template you must use, but it should look professional. You can use 1 or 2 columns per page. The contents of the report will vary depending on the type of project you are doing. However, it should certainly describe the goal of you project (what is your learning objective, or what problem are you trying to solve), your methodology, and some kind of evaluation of your results or progress. There are no hard limits on the length of your final report, but you should target something in the range of 5-10 pages.
Evaluation
Your final project will be evaluated according to the following criteria, with roughly equal weight placed on each one.
- Scope/Relevance (20%):Is the objective clear? Is the project course-related and substantial enough?
- Methodology (30%):Is the methodology appropriate and clearly described?
- Evaluation (30%):Did you evaluate your work? Did you achieve your objective? If not, did you explain why not?
- Presentation (20%):Is your report well organized and clearly written?
Your report should clearly indicate where you obtained any data that you used in your project. Include a link to the data if possible.
A note about "Evaluation": If you are implementing an algorithm (or several) in MapReduce/Spark, then evaluation will primarially be about determining the correctness of your implementation. (E.g. you should be testing your results to be sure that it actually works!) If you are learning a new framework then you should include a section on what you've learned. (You should also implement a simple algorithm to demonstrate what you've learned, and of course the results should be tested!).
The deadline for submission of your project report is 11pm on the last day of classes. As you are grad students I'm allowed to make this a "soft" deadline if you have a reason that you need more time. (This does not use a flex day! I will very likely say yes so if you need the time just ask!)