Exploring Distributed Systems: Security, File Systems, Overlay Networks, and Future Trends

Task:

For each item in each question, state your assumptions, for example what distributed hardware are you considering, migration of jobs is possible or not, what type of attack/fault model, etc.

1. Security and Attacks

a. Define DoS and DDoS attacks

b. Is it possible to completely solve the problem of DoS attacks? Why or why not?

c. Give 2 examples of what components of a distributed system can be attacked, and what the attacks would be, that is, the attack model.  State your assumptions about the distributed system you’re considering.

d. From one of your examples, present a solution to mitigation of DoS attacks.

2. Distributed file systems, Map Reduce and Distribution

a. Give an example of a problem that could use hadoop/map-reduce to solve a problem that is interesting to you.  Do not use the project 3 problem, nor other problems you find on the Internet, although you can feel free to read articles and find sources of inspiration (make sure you cite them).  Explain the problem characteristics that make hadoop/map-reduce appropriate.

b. Explain the best, most scalable, distributed architecture (including servers, file systems, network) to run map-reduce efficiently (clearly, you need to state your assumptions and define your metrics).

c. Compare and contrast Hadoop with other similar distributed systems, such as Apache Spark, Apache Storm, Google Big Query, Hydra, Ceph, Pachiderm, Apache Flink.  Pick one of these or any other system that is similar to Hadoop as a concrete example.

3. Distributed

a. Describe how overlay networks work, and the underlying network technology used in overlay networks.  Give details of where routing takes place, what you think the overheads are, where the overlay nodes are, who controls them

b. Describe the problems with reliability, availability, fault tolerance, and performance of overlay networks, and what is one possible solution for the problems you described.  Guide your answer by an example application for which you believe overlay networks would be useful. Do not use the examples in Dr. Babay’s slides and you can feel free to read articles and find sources of inspiration (make sure you cite them).

c. Compare and contrast, using a table, different cloud solutions PaaS, IaaS, SaaS, and any other “aaS” you can find.  In your table, give examples of applications that would be most suited for each of the types of solutions, and why.

d. Assume one can be extremely efficient in downloading and executing code to routers in the Internet.  With that, you may be able to solve a few problems, such as DoS, real-time scheduling in the network, billing different users for different classes of service, etc.  Pick a problem that you can solve with the new capability (download and execute code in network routers).

4. What is the next big thing in distributed systems?  That is, imagine something what will be hot in 5 years, and it doesn’t exist now, as a commercial system.