Assignments
Data-Intensive Distributed Computing (Fall 2021)
CS 451 Final Project
This project is for undergraduate students taking CS 451.
Introduction
In this project, we tackle the exciting problem of predicting the cryptocurrency market. Specifically, we will try to predict the price of Bitcoin (BTC) in the next minutes or hours. In traditional technical analysis, we utilize various metrics such as the relative strength index (RSI) to quantify market conditions. For example, they indicate if we are in an oversold or undervalued condition to predict future movements of Bitcoin price. These techniques rely on computing some metrics based on the historical market data. In this project, we are going to augment these techniques with the insight we obtain from Bitcoin blockchain data to improve their prediction power. Thanks to big data tools, we can obtain a much deeper insight into what is happening in the Bitcoin network. We are going to analyze the data in the Bitcoin blockchain to predict the future price of Bitcoin. But is this a big data problem? Bitcoin adds a new block to its blockchain about every 10 minutes. Each block is about 1 MB and the blockchain has more than 700,000 blocks. At the moment, the size of the entire Bitcoin blockchain is more than 370 GB and it is growing very fast. So it is indeed a big data problem which is getting more and more difficult.
Project description
In this project, you will utilize what you have learnt in this course to make a series of predictions about the Bitcoin future price. Specifically, you use big data frameworks such as Apache Spark or Apache Hadoop to predict how the market will change in the near future. The time window of the prediction can be minutes to a few hours. It is up to you to decide what time window to use in your evaluation. Your algorithm may only indicate the direction of movement (e.g., higher, lower, or no change), or specify the amount of change (e.g., +$240 in 15 minutes).
Here is a rough outline of what you should do:
- Download some recent blocks. You must use *raw* block data (i.e., download binary data and not processed data). You don’t need to download the entire blockchain. Start small and then test your algorithm on a bigger window of time. It is up to you to decide how many blocks you need for your estimator. Figuring out how to download Bitcoin blocks is part of this project. Find a suitable method and justify it.
- Covert the raw Bitcoin blocks to a file that contains the Bitcoin transactions. The data you need to write to this file should typically contain the sender and receiver addresses, the amount of Bitcoin, and the time of transaction. Feel free to include other information if it helps your algorithm. This step must be done by a big data framework so that it can be scaled to potentially the entire blockchain. Note that extracting transactions from the binary blocks is a bit complex. You need to understand how transactions are encoded in blocks. There are plenty of great Youtube videos on how the data looks like.
- Design a future price predictor. In this step, you need to come up with an idea to use the data we obtained in the last step to predict the future price of Bitcoin. Think about it for a while with your teammates and try to come up with an interesting idea. Don’t worry if it does not work at the end. This is a research project and it may fail. You can receive a full mark even for a failed project if you clearly explain your experience and the decisions you made and if you dig into reasons the project failed. In addition to the data we obtained in the last step, you can use any other data such as the historical price of Bitcoin. An example idea is trying to find specific patterns in transactions before the Bitcoin price jumps or crashes. Feel free to use any Big data framework for this step. For example, if you think a graph processing framework helps, you can use a framework like Spark GraphX.
- Evaluate the accuracy of your Bitcoin price predictor. Remove one or multiple blocks from your training set and try to predict the price for these blocks. Compare your predicted price with the actual price of bitcoin at that time.
Processing framework
You are allowed to use any big data framework or library. This is the time to utilize existing tools rather than reinventing them as we did in the assignments. Your project must use the Bitcoin blockchain data. Just fitting a line to the historical Bitcoin price is not acceptable. The prediction window is very small, but it does not mean you have to use streaming frameworks. You can implement and evaluate your algorithm on offline traces too.
Group size
This project can be done individually or in groups of up to 3 members. It is highly recommended to work in a group. You can use the Piazza's teammate search post to look for teammates. The amount of effort devoted to the project should be proportional to the number of people in the team. As a guideline, the level of effort should be comparable to two assignments per person.
Deliverables
The output of this project should be a short video up to 5 minutes (Note: longer videos will not be accepted even if it is 5:01!). In the video, explain (1) the idea, (2) methodology used, (3) implementation, and (4) results. You should design and implement an evaluation methodology that measures the accuracy of the results. For example, an algorithm can be trained on a subset of data and evaluated on the rest of the data. You must compare your algorithm’s predictions with the ground truth data.
Submission
Please submit your project here. Each group must submit only once.
Important node: This is a research project meaning that after spending a lot of time on an idea, it may still fail! This is perfectly acceptable for this project. You still need to present the 4 deliverable items in your video and conclude that idea X is not a suitable methodology for predicting the Bitcoin price. Finding out that a methodology does not work is also valuable in research. The amount of effort devoted to the project should be proportional to the number of people in the team. As a guideline, the level of effort should be comparable to two to three assignments per person.
The (hard) deadline for submission of your project report is 4 pm on Tuesday December 14, 2021.
CS 651 Final Project
This project is for graduate students taking CS 651.
Your project is the same as the CS451 project except in addition to the video you also submit a paper on what you have done in the project. Please see the deliverables below. You may work in groups of up to three, or you can also work by yourself if you wish. The amount of effort devoted to the project should be proportional to the number of people in the team. As a guideline, the level of effort should be comparable to two assignments per person.
Deliverable
The deliverables for the final project are a report and a video.
- Report: Use the ACM Templates. The contents of the report will vary depending on the type of project you are doing. However, it should certainly describe the goal of you project (what is your learning objective, or what problem are you trying to solve), your methodology, and some kind of evaluation of your results or progress. There are no hard limits on the length of your final report, but you should target something in the range of 4-10 pages.
- Video: In addition to your final report, you are required to deliver a short video up to 5 minutes (Note: longer videos will not be accepted even if it is 5:01!). This video must explain (1) the idea, (2) methodology used, (3) implementation, and (4) results.
The (hard) deadline for submission of your project report is 4 pm on Tuesday December 14, 2021.
Submission
Please submit your project here. Each group must submit only once.