Chat with us, powered by LiveChat   mongoDB,MapReduce and Spark {“AirCraft”:”A-1″,”AirCraftName”:”Douglas A-1 Skyraider”,”AirCraftType”:”Fighter Jet”} {“AirCraft”:”A-26″,”AirCraftName”:”Douglas A-26 Invader”,”AirCraftType”:”Ligh |

  mongoDB,MapReduce and Spark 

{“AirCraft”:”A-1″,”AirCraftName”:”Douglas A-1 Skyraider”,”AirCraftType”:”Fighter Jet”}
{“AirCraft”:”A-26″,”AirCraftName”:”Douglas A-26 Invader”,”AirCraftType”:”Light Bomber”}
{“AirCraft”:”A-37″,”AirCraftName”:”Cessna A-37 Dragonfly”,”AirCraftType”:”Light ground-attack aircraft”}
{“AirCraft”:”A-4″,”AirCraftName”:”McDonnell Douglas A-4 Skyhawk”,”AirCraftType”:”Fighter Jet”}
{“AirCraft”:”A-5″,”AirCraftName”:”North American A-5 Vigilante”,”AirCraftType”:”Bomber Jet”}
{“AirCraft”:”A-6″,”AirCraftName”:”Grumman A-6 Intruder”,”AirCraftType”:”Attack Aircraft”}
{“AirCraft”:”A-7″,”AirCraftName”:”LTV A-7 Corsair II”,”AirCraftType”:”Attack Aircraft”}
{“AirCraft”:”AC-119″,”AirCraftName”:”Fairchild AC-119 Shadow or Stinger”,”AirCraftType”:”Military Transport aircraft”}
{“AirCraft”:”AC-123″,”AirCraftName”:”Fairchild C-123 Provider “,”AirCraftType”:”Military Transport aircraft”}
{“AirCraft”:”AC-130″,”AirCraftName”:”Lockheed AC-130 Spectre”,”AirCraftType”:”Fixed wing ground attack gunship”}
{“AirCraft”:”AC-47″,”AirCraftName”:”Douglas AC-47 Spooky”,”AirCraftType”:”Ground attack aircraft”}
{“AirCraft”:”AH-1″,”AirCraftName”:”Bell AH-1 HueyCobra”,”AirCraftType”:”Helicopter”}
{“AirCraft”:”B-1″,”AirCraftName”:”Rockwell B-1 Lancer”,”AirCraftType”:”Heavy strategic bomber”}
{“AirCraft”:”B-52″,”AirCraftName”:”B-52 Stratofortress”,”AirCraftType”:”Strategic bomber”}
{“AirCraft”:”B-57″,”AirCraftName”:”Martin B-57 Canberra”,”AirCraftType”:”Tactical Bomber”}
{“AirCraft”:”B-66″,”AirCraftName”:”Douglas B-66 Destroyer”,”AirCraftType”:”Light Bomber”}
{“AirCraft”:”C-1″,”AirCraftName”:”Grumman C-1A Trader”,”AirCraftType”:”Transport”}
{“AirCraft”:”C-117″,”AirCraftName”:”C-117D Skytrain”,”AirCraftType”:”Transport”}
{“AirCraft”:”C-119″,”AirCraftName”:”Fairchild C-119 Flying Boxcar”,”AirCraftType”:”Military Transport aircraft”}
{“AirCraft”:”C-123″,”AirCraftName”:”Fairchild C-123 Provider “,”AirCraftType”:”Military Transport aircraft”}
{“AirCraft”:”C-130″,”AirCraftName”:”Lockheed C-130 Hercules”,”AirCraftType”:”Military Transport aircraft”}
{“AirCraft”:”C-47″,”AirCraftName”:”Douglas C-47 Skytrain”,”AirCraftType”:”Military Transport aircraft”}
{“AirCraft”:”C-54″,”AirCraftName”:”Douglas C-54 Skymaster”,”AirCraftType”:”Military Transport aircraft”}
{“AirCraft”:”C-7″,”AirCraftName”:”C-7A Caribou “,”AirCraftType”:”Military Transport aircraft”}
{“AirCraft”:”C-76″,”AirCraftName”:”Curtiss-Wright C-76 Caravan”,”AirCraftType”:”Medium Transport”}
{“AirCraft”:”CH-46″,”AirCraftName”:”Boeing Vertol CH-46 Sea Knight”,”AirCraftType”:”Cargo Helicopter”}
{“AirCraft”:”CH-53″,”AirCraftName”:”Sikorsky CH-53 Sea Stallion”,”AirCraftType”:”Heavy-lift cargo helicopter”}
{“AirCraft”:”CH-3″,”AirCraftName”:”Sikorsky CH-3E”,”AirCraftType”:”Heavy-lift cargo helicopter”}
{“AirCraft”:”CH-47″,”AirCraftName”:”Boeing CH-47 Chinook”,”AirCraftType”:”Transport Helicopter”}
{“AirCraft”:”DC-4″,”AirCraftName”:”Douglas DC-4″,”AirCraftType”:”Airliner/transport aircraft”}

DS 5110: Introduction to Data Management and Processing (Fall 2022)
Student name:

MongoDB and Big Data

1 MongoDB

Download the file restaurants.json from Canvas and load it into a MongoDB database. The
file contains 3772 documents.

Write the following MongoDB queries on the restaurants collection (you can write them
either directly in the MongoDB shell or inside a python script):

1. Display all the restaurants located in the boroughs Bronx or Brooklyn.

2. Find the restaurant id, name, borough and cuisine for those restaurants whose name
starts with the letters ’Mad’.

3. Find the restaurants that have received a score between 80 and 90.

4. Display the restaurant id and name of restaurants which have received ’C’ grade in year

5. Find how many restaurants belong to each cuisine (note that the cuisine attribute may
contain more than one cuisine type, e.g., ”Ice Cream, Gelato, Yogurt, Ices”).

6. Find the restaurants that do not prepare any cuisine of ’American’ and their average
grade score is higher than 30. Display the restaurant ids and their average score.

7. For each restaurant display only the grades that were recorded from the year 2014

8. Calculate the average score across all the restaurants in the collection.


Dr. Roi Yehoshua
(Due) November 20, 2022

2 MapReduce

Describe how to implement the following relational operations using MapReduce. Write the
map and reduce functions in pseudocode.

1. Projection πS(R): From each tuple of relation R produce only the components for the
attributes in S.

2. Intersection R∩S: Return the tuples that are present in both relations R and S. Assume
that relations R and S have the same schema (same attributes and same type).

3. Grouping γA,θ(B)(R). Given a relation R(A,B,C), with one grouping attribute A, one
aggregated attributeB, and another attribute C, which is neither grouped or aggregated:

(a) Partition the tuples of R according to their values in attribute A.

(b) For each group, aggregate the values in attribute B and apply function θ on the
aggregated value (θ is an aggregation operation such as SUM, COUNT or MAX).

The result of this operation is one tuple for each group. That tuple has a component
for the grouping attribute A, with the value common to tuples of that group. It also
has a component for each aggregation θ(B), with the aggregated value for that group.


3 Spark

The bombing campaigns of the Vietnam War were the longest and heaviest aerial bombard-
ment in history. The following datasets describe all the air force operations during the Vietnam

The file Bombing_Operations.json contains the following attributes:

• AirCraft: Aircraft model (example: EC-47)
• Cont

{“address”: {“building”: “1007”, “coord”: [-73.856077, 40.848447], “street”: “Morris Park Ave”, “zipcode”: “10462”}, “borough”: “Bronx”, “cuisine”: “Bakery”, “grades”: [{“date”: {“$date”: 1393804800000}, “grade”: “A”, “score”: 2}, {“date”: {“$date”: 1378857600000}, “grade”: “A”, “score”: 6}, {“date”: {“$date”: 1358985600000}, “grade”: “A”, “score”: 10}, {“date”: {“$date”: 1322006400000}, “grade”: “A”, “score”: 9}, {“date”: {“$date”: 1299715200000}, “grade”: “B”, “score”: 14}], “name”: “Morris Park Bake Shop”, “restaurant_id”: “30075445”}
{“address”: {“building”: “469”, “coord”: [-73.961704, 40.662942], “street”: “Flatbush Avenue”, “zipcode”: “11225”}, “borough”: “Brooklyn”, “cuisine”: “Hamburgers”, “grades”: [{“date”: {“$date”: 1419897600000}, “grade”: “A”, “score”: 8}, {“date”: {“$date”: 1404172800000}, “grade”: “B”, “score”: 23}, {“date”: {“$date”: 1367280000000}, “grade”: “A”, “score”: 12}, {“date”: {“$date”: 1336435200000}, “grade”: “A”, “score”: 12}], “name”: “Wendy’S”, “restaurant_id”: “30112340”}
{“address”: {“building”: “351”, “coord”: [-73.98513559999999, 40.7676919], “street”: “West 57 Street”, “zipcode”: “10019”}, “borough”: “Manhattan”, “cuisine”: “Irish”, “grades”: [{“date”: {“$date”: 1409961600000}, “grade”: “A”, “score”: 2}, {“date”: {“$date”: 1374451200000}, “grade”: “A”, “score”: 11}, {“date”: {“$date”: 1343692800000}, “grade”: “A”, “score”: 12}, {“date”: {“$date”: 1325116800000}, “grade”: “A”, “score”: 12}], “name”: “Dj Reynolds Pub And Restaurant”, “restaurant_id”: “30191841”}
{“address”: {“building”: “2780”, “coord”: [-73.98241999999999, 40.579505], “street”: “Stillwell Avenue”, “zipcode”: “11224”}, “borough”: “Brooklyn”, “cuisine”: “American “, “grades”: [{“date”: {“$date”: 1402358400000}, “grade”: “A”, “score”: 5}, {“date”: {“$date”: 1370390400000}, “grade”: “A”, “score”: 7}, {“date”: {“$date”: 1334275200000}, “grade”: “A”, “score”: 12}, {“date”: {“$date”: 1318377600000}, “grade”: “A”, “score”: 12}], “name”: “Riviera Caterer”, “restaurant_id”: “40356018”}
{“address”: {“building”: “97-22”, “coord”: [-73.8601152, 40.7311739], “street”: “63 Road”, “zipcode”: “11374”}, “borough”: “Queens”, “cuisine”: “Jewish/Kosher”, “grades”: [{“date”: {“$date”: 1416787200000}, “grade”: “Z”, “score”: 20}, {“date”: {“$date”: 1358380800000}, “grade”: “A”, “score”: 13}, {“date”: {“$date”: 1343865600000}, “grade”: “A”, “score”: 13}, {“date”: {“$date”: 1323907200000}, “grade”: “B”, “score”: 25}], “name”: “Tov Kosher Kitchen”, “restaurant_id”: “40356068”}
{“address”: {“building”: “8825”, “coord”: [-73.8803827, 40.7643124], “street”: “Astoria Boulevard”, “zipcode”: “11369”}, “borough”: “Queens”, “cuisine”: “American “, “grades”: [{“date”: {“$date”: 1416009600000}, “grade”: “Z”, “score”: 38}, {“date”: {“$date”: 1398988800000}, “grade”: “A”, “score”: 10}, {“date”: {“$date”: 1362182400000}, “grade”: “A”, “score”: 7}, {“date”: {“$date”: 1328832000000}, “grade”: “A”, “score”: 13}], “name”: “Brunos On The Boulevard”, “restaura