About this previous year paper
SECTION A
(Attempt all | 2 × 10 = 20 Marks)
(a) Benefits of HDFS over NFS
HDFS provides high fault tolerance, scalability, and distributed storage across multiple nodes, whereas NFS is centralized and less reliable for large-scale data processing. HDFS is optimized for big data analytics and parallel processing.
(b) Structured, Semi-Structured & Unstructured Data
Structured data follows a fixed schema (tables, rows). Semi-structured data uses tags or markers (XML, JSON). Unstructured data has no predefined format, such as images, videos, and social media data.
(c) Sources of data in Big Data
Sources include social media, sensors and IoT devices, transaction logs, web clickstreams, mobile devices, multimedia content, and enterprise applications.
(d) Metadata in HDFS
Metadata stores information about files such as file name, size, permissions, block locations, and replication details. It is maintained by the NameNode.
(e) Map vs Reduce
Map processes input data and converts it into key-value pairs. Reduce aggregates and processes these key-value pairs to produce final output.
(f) Indexing
Indexing is a technique used to improve data retrieval speed by creating a data structure that allows quick access to records.
(g) Shuffle vs Sort
Shuffle transfers intermediate map outputs to reducers. Sort arranges the data by keys before reduction.
(h) TF-IDF
TF-IDF (Term Frequency–Inverse Document Frequency) measures the importance of a word in a document relative to a collection of documents.
(i) NameNode, DataNode, JobTracker, TaskTracker
NameNode manages metadata, DataNode stores data blocks, JobTracker manages MapReduce jobs, and TaskTracker executes tasks on slave nodes.
(j) File name and block size
Windows: Max filename 255 chars, block size ~4 KB
Linux: Max filename 255 chars, block size ~4 KB
Hadoop: Block size 128 MB (default), large block size for efficient processing
SECTION B
(Attempt any THREE | 10 × 3 = 30 Marks)
2(a) 5 Vs of Big Data and their importance
The 5 Vs are Volume (huge data size), Velocity (speed of data generation), Variety (multiple data types), Veracity (data quality), and Value (useful insights). These characteristics explain why traditional systems fail and why specialized Big Data tools are required.
2(b) History and evolution of Hadoop
Hadoop originated from Google’s GFS and MapReduce papers. It evolved into Apache Hadoop, providing open-source distributed storage (HDFS) and processing (MapReduce), later enhanced with YARN, Spark, Hive, and HBase.
2(c) Data replication in HDFS
Replication stores multiple copies of data blocks across different nodes. Benefits include fault tolerance and high availability. Challenges include increased storage cost and network overhead.
2(d) Fair vs Capacity Scheduler in YARN
Fair Scheduler ensures equal resource sharing among applications, while Capacity Scheduler allocates fixed resources to queues. Fair Scheduler is flexible; Capacity Scheduler is suitable for large organizations.
2(e) Pig and its execution modes
Apache Pig uses Pig Latin for data processing.
Execution modes: Local Mode MapReduce Mode
Pig simplifies complex data flows compared to traditional databases, which use SQL and structured schema.
SECTION C
3(a) Security, compliance, auditing & protection in Big Data
Big Data security includes authentication, authorization, encryption, auditing, and compliance. Key features are data privacy, secure access control, regulatory compliance, and monitoring using tools like Kerberos and Ranger.
3(b) Challenges of conventional data systems
Traditional systems lack scalability, flexibility, and performance for large datasets. Big Data solves these using distributed storage, parallel processing, and fault tolerance.
4(a) Hadoop Distributed File System
HDFS stores large data across clusters using replication and parallel access. It supports scalability, fault tolerance, and high-throughput processing.
4(b) Anatomy of a MapReduce job
Input split → Map → Shuffle → Sort → Reduce → Output. JobTracker coordinates tasks while TaskTrackers execute them.
5(a) Data ingestion methods: Flume & Sqoop
Flume ingests streaming data like logs, while Sqoop transfers structured data between RDBMS and Hadoop.
5(b) Hadoop I/O support
Hadoop supports compression, serialization, Avro for schema-based storage, and file formats like SequenceFiles and Parquet.
6(a) NoSQL & MongoDB
MongoDB stores data as documents using JSON-like format. It supports CRUD operations, flexible schema, indexing, and high scalability.
6(b) Scala features
Scala supports object-oriented and functional programming, classes, objects, closures, pattern matching, and is tightly integrated with Spark.
7(a) HBase vs RDBMS
HBase is distributed, schema-less, and scalable, while RDBMS is centralized and schema-based. HBase offers advanced indexing and column-family design.
7(b) Role of ZooKeeper
ZooKeeper manages configuration, synchronization, leader election, and monitoring in Hadoop clusters. It helps build reliable distributed applications.