📊 Technology · Banking Exam 2025
Big Data Complete
Exam Guide
5 Vs · Hadoop Ecosystem · Apache Spark · NoSQL · Batch vs Stream · Analytics Types · Banking Applications · India Initiatives · Cloud Platforms · 50 MCQs with Answers — GATE, IBPS, RBI, UPSC, SSC
GATE CSIBPS PORBI Grade BUPSC GS-III
SSC CGLNABARDBank InternalCampus Placements
📋 Table of Contents
1. What is Big Data?
2. The 5 Vs of Big Data
3. Types of Data
4. Hadoop Ecosystem
5. Apache Spark
6. NoSQL Databases
7. Batch vs Stream Processing
8. Analytics Types
9. Big Data in Banking
10. India Initiatives
11. Cloud & Big Data
12. Security & Governance
13. MCQ Practice (50 Qs)
14. Quick Revision Cards
📊
Foundation — Start HereWhat is Big Data?
Big Data refers to extremely large, complex, and fast-growing datasets that cannot be managed, processed, or analysed using traditional database systems like Excel or SQL alone. It requires specialised tools, architectures, and technologies to extract meaningful insights.
⚡ Core Exam Facts
- Defined by 5 Vs: Volume, Velocity, Variety, Veracity, Value
- Originally 3 Vs (Gartner, 2001): Volume, Velocity, Variety — then 2 more added
- Core framework: Hadoop (HDFS + MapReduce + YARN)
- Hadoop was originally developed by Yahoo! (inspired by Google’s GFS & MapReduce papers)
- Hadoop is written in Java; Spark is written in Scala (also supports Python, Java, R)
- Big Data ≠ just “large data” — it’s about data that cannot be handled by traditional tools
| Data Source | Data Type | Example |
|---|---|---|
| Banking transactions | Structured | Account number, amount, balance |
| UPI/mobile payments | Structured | Transaction ID, timestamp, merchant |
| WhatsApp/social media | Unstructured | Text, audio, video, images |
| IoT sensors (ATMs) | Semi-structured | Temperature logs, alerts (JSON/XML) |
| Customer complaints | Unstructured | Free-form text, emails |
| Market data feeds | Structured | Stock prices, order books |
5️⃣
Most Frequently Tested — Know All 5The 5 Vs of Big Data
V
Volume
Size / amount of data generated
Millions of ATM & UPI transactions per day
V
Velocity
Speed of data generation & processing
Real-time UPI transactions per second
V
Variety
Different forms: structured, semi, unstructured
Excel sheets, images, JSON, tweets, videos
V
Veracity
Accuracy & trustworthiness of data
Removing duplicate or incorrect entries
V
Value
Usefulness / business benefit extracted
Fraud detection, risk prediction, insights
🎯 5 Vs — Exam Key Points
- Original 3 Vs (Gartner 2001) = Volume, Velocity, Variety | 2 added later = Veracity, Value
- NOT one of the 5 Vs: Visibility, Validity, Variability, Visualization — common wrong answer options
- Volume = how much | Velocity = how fast | Variety = how diverse
- Veracity = how accurate/trustworthy | Value = how useful (business impact)
- “What are the 3 Vs?” = Volume, Velocity, Variety (NOT Veracity or Value)
📁
Always TestedTypes of Data in Big Data
| Type | Description | Format | Banking Example | % of All Data |
|---|---|---|---|---|
| Structured | Organised in rows and columns with fixed schema | Tables, CSV, RDBMS | Core banking — account no, balance, transactions | ~10% |
| Semi-Structured | Not fixed format but has tags/markers for organisation | XML, JSON, Email headers | API responses, IoT sensor data, UPI alerts | ~20% |
| Unstructured | Raw, no predefined format — hardest to process | Text, audio, video, images | Customer complaints, social media, call recordings | ~70% |
📌 Data Types — Exam Key Points
- ~80% of world’s data is unstructured — emails, videos, social media posts
- Traditional RDBMS (MySQL, Oracle) can only handle structured data efficiently
- NoSQL databases handle all three types — especially unstructured and semi-structured
- HDFS can store all three types — that’s its key advantage over traditional file systems
- JSON and XML = semi-structured (have tags but no fixed schema)
🐘
Most Tested Big Data TopicHadoop Ecosystem
Apache Hadoop is an open-source framework for distributed storage and parallel processing of large datasets across clusters of computers. Created by Doug Cutting (inspired by Google’s GFS and MapReduce papers), developed at Yahoo!. Written in Java. The elephant logo was named after Doug Cutting’s son’s toy elephant.
Core — Storage
HDFS
Hadoop Distributed File System
Stores huge data across multiple DataNodes. NameNode manages metadata. Default block size = 128 MB. Data replicated 3x for fault tolerance.
Storage foundation of Hadoop
Core — Processing
MapReduce
Map → Shuffle → Reduce
Map: splits data, processes into key-value pairs. Reduce: aggregates intermediate results. Batch processing. Disk-based (slower than Spark).
Parallel batch processing engine
Core — Resources
YARN
Yet Another Resource Negotiator
Manages and schedules cluster resources. Acts as the Operating System of Hadoop. Replaced old JobTracker/TaskTracker in Hadoop 2.x.
Resource management layer
Query Tool
Hive
SQL-like queries on Hadoop
Provides HiveQL — SQL-like language for querying structured data in HDFS. Translates queries to MapReduce jobs. Used by analysts.
SQL on Big Data
ETL Tool
Pig
Pig Latin scripting language
Data flow scripting and ETL (Extract-Transform-Load) operations. Uses Pig Latin language. Good for data transformation pipelines.
Data transformation / ETL
NoSQL DB
HBase
Hadoop Database
Column-oriented NoSQL database built on HDFS. Handles billions of rows and millions of columns. Real-time read/write access.
NoSQL database on HDFS
Data Transfer
Sqoop
SQL-to-Hadoop
Transfers data between Hadoop and RDBMS (MySQL, Oracle). Import/export in bulk. Key integration tool.
RDBMS ↔ Hadoop transfer
Log Collector
Flume
Flume data collector
Collects, aggregates, and moves streaming log data (server logs, clickstreams) to HDFS. Reliable and distributed.
Log data collection to HDFS
Scheduler
Oozie
Hadoop Workflow Scheduler
Schedules and manages Hadoop job workflows. Runs jobs in the correct order and at the right times (cron-like scheduling).
Workflow scheduling for Hadoop
Machine Learning
Mahout
Apache Mahout
Provides scalable ML algorithms — clustering, classification, recommendations — that run on Hadoop/Spark.
ML algorithms on Hadoop
| Component | One-liner | Exam Trigger Words |
|---|---|---|
| HDFS | Distributed file storage across cluster | “stores data”, “distributed storage”, “128 MB blocks” |
| MapReduce | Parallel batch processing (Map + Reduce) | “parallel processing”, “key-value pairs”, “batch” |
| YARN | Resource management and scheduling | “resource manager”, “cluster management”, “scheduler” |
| Hive | SQL-like querying (HiveQL) | “SQL on Hadoop”, “query tool”, “HiveQL” |
| Pig | ETL / data transformation scripting | “ETL”, “data flow”, “Pig Latin” |
| HBase | NoSQL column-store on HDFS | “NoSQL in Hadoop”, “column-oriented”, “real-time read/write” |
| Sqoop | Transfer between RDBMS and Hadoop | “RDBMS to Hadoop”, “import/export”, “SQL + Hadoop” |
| Flume | Collect and move streaming logs to HDFS | “log data”, “streaming data collection”, “clickstream” |
| Oozie | Workflow scheduler for Hadoop jobs | “schedule”, “workflow”, “job sequencing” |
| Mahout | Machine Learning on Hadoop | “ML”, “recommendation”, “clustering on Hadoop” |
📌 Hadoop Architecture — HDFS Details
- NameNode = master node; stores metadata (directory tree, file locations) — does NOT store data
- DataNode = stores actual data blocks; reports to NameNode
- Default block size = 128 MB (Hadoop 2.x) — was 64 MB in Hadoop 1.x
- Data replication factor = 3 copies by default (fault tolerance)
- Hadoop NOT part of core: Cassandra is NOT a Hadoop component (it’s a separate NoSQL DB)
⚡
Faster than MapReduce — Key Exam TopicApache Spark
Apache Spark is an open-source, in-memory data processing engine that is up to 100x faster than Hadoop MapReduce for in-memory operations. Unlike MapReduce, which writes intermediate results to disk, Spark keeps data in RAM. Written in Scala but supports Python (PySpark), Java, and R. Can work with or without Hadoop.
| Feature | Hadoop MapReduce | Apache Spark |
|---|---|---|
| Processing | Batch only | Batch + Real-time stream + Interactive |
| Speed | Slower (disk I/O) | Up to 100x faster (in-memory) |
| Data storage | Writes to disk at each step | Keeps in RAM (in-memory computing) |
| Ease of use | Complex Java code | Simpler APIs (Python, Scala, Java, R) |
| Language | Java primarily | Scala + Python, Java, R |
| Machine Learning | Mahout (limited) | MLlib (built-in, powerful) |
| Real-time | Not supported | Spark Streaming (near real-time) |
| Graph processing | Limited | GraphX (built-in) |
✅ Spark Components
- Spark Core = base engine — RDD (Resilient Distributed Dataset) for fault-tolerant parallel data
- Spark SQL = SQL queries on structured data using DataFrames
- Spark Streaming = near real-time stream processing (micro-batch)
- MLlib = built-in Machine Learning library (classification, clustering, regression)
- GraphX = graph analytics and computation
- Banking uses: fraud detection, credit scoring, sentiment analysis, real-time transaction monitoring
🗄️
Non-Relational DatabasesNoSQL Databases
NoSQL (Not Only SQL) databases are designed to handle large volumes of unstructured and semi-structured data with horizontal scalability. Unlike RDBMS (which uses fixed schemas and tables), NoSQL uses flexible data models.
| Feature | RDBMS (SQL) | NoSQL |
|---|---|---|
| Data model | Tables and rows (fixed schema) | Document, key-value, column, graph |
| Schema | Fixed — defined upfront | Flexible (schema-on-read) |
| Scalability | Vertical (bigger machine) | Horizontal (add more machines) |
| Data type | Structured only | Structured, semi-structured, unstructured |
| Consistency | ACID (strong) | BASE (eventually consistent) |
| Examples | MySQL, Oracle, PostgreSQL | MongoDB, Cassandra, HBase, CouchDB, Redis |
| Best for | Transactional banking (core CBS) | Big Data, social, real-time analytics |
| NoSQL Type | Example | Use Case | Key Feature |
|---|---|---|---|
| Document Store | MongoDB | User profiles, product catalogs | Stores JSON-like documents |
| Column Store | HBase, Cassandra | Time-series, analytics, IoT data | High write throughput |
| Key-Value Store | Redis, DynamoDB | Caching, session management | Fastest read/write |
| Graph Database | Neo4j | Fraud networks, social relationships | Node-relationship model |
🎯 CAP Theorem — Exam Favourite
- CAP Theorem = Consistency, Availability, Partition Tolerance — a distributed system can guarantee only 2 of 3
- Consistency = all nodes see the same data at the same time
- Availability = system always responds to requests
- Partition Tolerance = system works even if network partition occurs
- MongoDB = CP | Cassandra = AP | Traditional RDBMS = CA (no partition tolerance)
🌊
Key DistinctionBatch Processing vs Stream Processing
📦 Batch Processing
- Processes data in bulk, at scheduled times
- High latency (minutes to hours)
- Cost-effective for large volumes
- Tool: Hadoop MapReduce, Hive
- Example: Monthly salary processing, end-of-day bank reports
- Example: Generating overnight risk reports for RBI
⚡ Stream / Real-time Processing
- Processes data continuously as it arrives
- Very low latency (milliseconds to seconds)
- More complex and expensive
- Tool: Spark Streaming, Apache Kafka, Flink, Storm
- Example: Real-time UPI fraud detection, live stock prices
- Example: Instant credit card transaction scoring
| Tool | Type | Use Case | Key Feature |
|---|---|---|---|
| Hadoop MapReduce | Batch | Large-scale batch analytics | Parallel disk-based processing |
| Apache Spark | Batch + Stream | Fast analytics, ML, real-time | In-memory, 100x faster than MR |
| Apache Kafka | Streaming (messaging) | Real-time data pipelines, event streaming | High-throughput message queue |
| Apache Flink | Stream | True real-time stream analytics | Event-time processing, stateful |
| Apache Storm | Stream | Real-time stream processing | Fault-tolerant, low latency |
| Spark Streaming | Micro-batch (near real-time) | UPI fraud detection, live analytics | Built into Spark ecosystem |
📈
4 Types — All Exam FavouritesBig Data Analytics Types
ANSWERS: “What happened?”
Descriptive Analytics
Summarises historical data to understand what happened in the past. Simplest and most common type.
Banking: Monthly transaction reports, NPA summary dashboard
ANSWERS: “Why did it happen?”
Diagnostic Analytics
Examines data to understand the root cause of events. Goes deeper than descriptive to explain WHY.
Banking: Fraud reason analysis, why branch revenue fell
ANSWERS: “What will happen?”
Predictive Analytics
Uses statistical models and ML to forecast future events. Most valuable for risk management.
Banking: Predicting loan default risk, NPA prediction
ANSWERS: “What should be done?”
Prescriptive Analytics
Recommends specific actions to take based on data and predictions. Most advanced analytics type.
Banking: Recommending loan terms, personalised product offers
🎯 Analytics — Exam Memory Order
- Order: Descriptive → Diagnostic → Predictive → Prescriptive (past to future to action)
- Descriptive = What happened? (PAST) | Diagnostic = Why? (PAST cause)
- Predictive = What will happen? (FUTURE) | Prescriptive = What should I do? (ACTION)
- Most complex and valuable = Prescriptive | Most common = Descriptive
🏦
Very High Exam WeightBig Data in Banking & Finance
🔍 Fraud Detection
Real-time analysis of every transaction using ML models. Detects unusual patterns instantly. Blocks suspicious transactions before completion.
Tech: Spark Streaming + ML | HDFC EVA, SBI analytics
📊 Credit Scoring
Uses traditional + alternate data (mobile usage, digital behaviour, utility payments) for more accurate and inclusive credit risk assessment.
Tech: Predictive Analytics + ML models
🛡️ AML / Compliance
Detects money laundering patterns — structuring, layering, smurfing. Maintains audit trails and data lineage for regulatory compliance.
Tech: Graph analytics + Rule-based + ML
👥 Customer Segmentation
Groups customers by behaviour, demographics, risk profile for targeted marketing, personalised products, and cross-selling.
Tech: K-Means clustering + Descriptive Analytics
⚠️ NPA Prediction
Predicts which loan accounts will become NPAs months before default. Enables proactive recovery action and provisioning decisions.
Tech: Predictive Analytics + XGBoost
🎯 Personalised Offers
Recommends credit cards, loans, FDs, insurance based on individual customer transaction history and life events (salary increase, new home).
Tech: Prescriptive Analytics + Recommendation engines
📋 Regulatory Reporting
Automates data collection and report generation for RBI/SEBI/IRDAI. Maintains complete audit trails. Reduces manual compliance effort.
Tech: Data warehousing + ETL pipelines
📈 Market Surveillance
SEBI uses Big Data to detect insider trading, market manipulation, and abnormal trading patterns in real time across all listed securities.
Tech: Stream processing + Pattern detection
| Institution | Big Data Use |
|---|---|
| RBI | Data analytics for fraud monitoring, AML, risk reporting, MuleHunter.AI |
| HDFC Bank | EVA chatbot + customer analytics + real-time fraud detection |
| SBI | Big Data platform for spending pattern analysis, fraud detection |
| SEBI | Market surveillance for insider trading, abnormal volume detection |
| NPCI | UPI fraud risk scoring in real time using Big Data + ML |
| NABARD | Credit and rural data analytics for agricultural lending decisions |
🇮🇳
Current Affairs — High 2025 Exam ProbabilityIndia’s Big Data & Data Initiatives
🇮🇳 Key Indian Big Data Initiatives
NDAP
National Data and Analytics Platform — NITI Aayog’s unified public data portal
MeghRaj
GI Cloud (Government of India Cloud) — cloud infrastructure for data storage
UIDAI / Aadhaar
One of world’s largest Big Data projects — 1.4 billion biometric records
IndiaAI Mission
₹10,372 crore — includes Big Data infrastructure for AI (2024)
| Initiative | By Whom | Purpose | Exam Relevance |
|---|---|---|---|
| NDAP | NITI Aayog | Unified platform for government data discovery and analytics | ⭐⭐⭐⭐⭐ |
| MeghRaj (GI Cloud) | MeitY | Government cloud for storing and processing large datasets | ⭐⭐⭐⭐⭐ |
| UIDAI / Aadhaar | UIDAI / Govt. | World’s largest biometric Big Data project — 1.4 billion records | ⭐⭐⭐⭐⭐ |
| Digital India | Govt. of India | Creates massive digital data footprint for Big Data analytics | ⭐⭐⭐⭐ |
| RBI IT Framework (2017) | RBI | Data governance, analytics guidelines for banks and NBFCs | ⭐⭐⭐⭐ |
| IndiaAI Mission | Govt. (2024) | ₹10,372 crore — builds AI + Big Data compute infrastructure | ⭐⭐⭐⭐⭐ |
| NITI Aayog “AI for All” | NITI Aayog | National AI strategy that promotes Big Data + AI in governance | ⭐⭐⭐⭐ |
☁️
Cloud Platforms for Big DataCloud & Big Data Integration
| Cloud Provider | Big Data Service | What It Does |
|---|---|---|
| AWS (Amazon) | Amazon EMR | Managed Hadoop/Spark cluster service |
| AWS | Amazon Redshift | Fully managed cloud data warehouse for analytics |
| AWS | Amazon S3 | Object storage — most popular cloud storage for Big Data |
| Azure (Microsoft) | Azure HDInsight | Managed Hadoop/Spark/Hive clusters on Azure |
| Azure | Azure Synapse Analytics | Integrated analytics service (warehouse + Big Data) |
| Google Cloud | BigQuery | Serverless, scalable data warehouse — SQL queries on petabytes |
| Google Cloud | Dataproc | Managed Hadoop and Spark service on Google Cloud |
📌 Key Cloud Big Data Facts for Exams
- Amazon Redshift = cloud data warehouse by AWS (NOT real-time streaming)
- Google BigQuery = serverless SQL queries on petabyte-scale data
- AWS S3 = Simple Storage Service — most widely used cloud object storage for Big Data
- Azure Synapse = combines data warehouse + Big Data analytics in one service
- Data Lake vs Data Warehouse: Lake = raw unprocessed | Warehouse = processed, structured, query-ready
🔐
Governance & PrivacyBig Data Security & Governance
| Term | Meaning | Banking Relevance |
|---|---|---|
| Data Governance | Framework of policies for data quality, access control, and usage | RBI IT Framework mandate for banks |
| Data Lineage | Tracking origin and movement of data through systems | Audit trails for regulatory compliance |
| Data Privacy | Protecting customer PII (Personally Identifiable Information) | India DPDP Act 2023, GDPR compliance |
| Encryption | Converting data to unreadable format — at rest and in transit | Protecting transaction and customer data |
| Anonymisation | Removing/masking personal identifiers from datasets | Using customer data for analytics without exposing PII |
| Data Cleansing | Removing duplicate, incorrect, or incomplete records | Ensuring veracity of credit data |
| Data Lake | Central repository of raw data in native format for later use | Banks store all transaction logs for future analytics |
| Data Swamp | Poorly governed Data Lake — full of ungoverned, unusable data | Risk when governance is not applied to data lakes |
| ETL | Extract, Transform, Load — process of moving and preparing data | Moving data from CBS to analytics platform |
📝
Tap Any Option to Reveal AnswerMCQ Practice — 50 Questions
CH.1Big Data Basics & 5 VsQ.01–Q.12
The original “3 Vs” of Big Data are:
✔ Correct: A — Volume, Velocity, Variety
The original 3 Vs coined by Gartner (2001): Volume (size), Velocity (speed), Variety (types). Two more added later: Veracity (accuracy) and Value (usefulness). If asked “3 Vs” — do NOT include Veracity or Value.Which of the following is NOT a characteristic (V) of Big Data?
✔ Correct: D — Visibility is NOT a V of Big Data
The 5 Vs are: Volume, Velocity, Variety, Veracity, Value. Visibility is NOT one of them. Common distractors in exams: Visibility, Validity, Variability, Visualization — none of these are the official 5 Vs.“Veracity” in the context of Big Data refers to:
✔ Correct: C — Accuracy and trustworthiness
Veracity = reliability, accuracy, and quality of data. Addresses uncertainty, inconsistency, and noise in data. Removing duplicate/incorrect entries improves veracity. Value = business benefit; Velocity = speed.Which statement about Big Data is TRUE?
✔ Correct: A
The defining characteristic of Big Data = traditional systems (Excel, SQL databases) cannot handle it efficiently. Requires specialised tools like Hadoop, Spark, NoSQL. Comes from multiple sources and has high velocity.Which type of data is represented by customer emails, audio recordings, and social media posts?
✔ Correct: C — Unstructured data
Unstructured data has no predefined format — emails, audio/video, social media posts, images. Makes up ~70–80% of all data. Cannot be stored in traditional rows-and-columns format. Requires NLP and ML to process.JSON and XML files in Big Data are examples of:
✔ Correct: B — Semi-structured data
Semi-structured data has tags/markers for organisation but no rigid fixed schema — JSON, XML, email headers, log files, IoT sensor data. It’s between structured (tables) and unstructured (free-form text).Hadoop was originally developed at which company?
✔ Correct: A — Yahoo!
Hadoop was created by Doug Cutting (named after his son’s toy elephant) while working at Yahoo!. It was inspired by Google’s published papers on Google File System (GFS) and MapReduce. Written in Java.Hadoop is primarily written in which programming language?
✔ Correct: A — Java
Hadoop = Java. Spark = Scala (but supports Python/Java/R). This distinction is frequently tested. Also: MapReduce jobs are written in Java; PySpark uses Python for Spark.The two Vs added to the original “3 Vs” to make it “5 Vs” are:
✔ Correct: A — Veracity and Value
3 Vs (original) = Volume, Velocity, Variety. Two more added = Veracity (accuracy/trustworthiness) and Value (business benefit). So 5 Vs total.Which data size unit comes after Terabyte?
✔ Correct: C — Petabyte
Data size order: KB → MB → GB → TB → PB (Petabyte) → EB (Exabyte) → ZB (Zettabyte). 1 Petabyte = 1,024 Terabytes. Big Data often measured in Petabytes and Exabytes.“Predictive Analytics” in Big Data answers which question?
✔ Correct: C — What will happen next?
Analytics order: Descriptive (What happened?) → Diagnostic (Why?) → Predictive (What will happen?) → Prescriptive (What should be done?). Predictive uses ML models to forecast future events.Which analytics type provides recommended ACTIONS based on data?
✔ Correct: D — Prescriptive Analytics
Prescriptive Analytics = most advanced type. Recommends specific actions to take. Example: “Offer this customer a home loan at 8.5% to maximise acceptance probability.” Most valuable but also most complex.CH.2Hadoop Ecosystem ComponentsQ.13–Q.26
HDFS stands for:
✔ Correct: A
HDFS = Hadoop Distributed File System. The storage layer of Hadoop. Splits large files into blocks (default 128 MB), distributes across DataNodes, replicates 3x for fault tolerance. NameNode stores metadata.The two CORE components of Hadoop are:
✔ Correct: A — HDFS and MapReduce
The two foundational components of Hadoop: HDFS (storage) and MapReduce (processing). YARN was added in Hadoop 2.x as the resource manager. All other tools (Hive, Pig, HBase) are part of the ecosystem built on top.YARN stands for:
✔ Correct: A — Yet Another Resource Negotiator
YARN = resource management layer of Hadoop 2.x. Manages CPU, memory, and scheduling across the cluster. Think of it as the “OS” of Hadoop — allocates resources to different jobs.Which Hadoop component provides SQL-like querying capability on Big Data?
✔ Correct: A — Hive
Hive = “SQL on Big Data.” Uses HiveQL (similar to SQL) to query data stored in HDFS. Translates queries into MapReduce jobs. Used by business analysts who know SQL but not Java.Apache Sqoop is used for:
✔ Correct: A — Transferring data between RDBMS and Hadoop
Sqoop = SQL + Hadoop. Imports data from relational databases (MySQL, Oracle, PostgreSQL) INTO Hadoop HDFS, and exports data from HDFS back to RDBMS. Key integration bridge.Apache Flume is primarily used for:
✔ Correct: B — Collecting streaming log data to HDFS
Flume = collects, aggregates, and moves streaming event/log data (server logs, clickstreams, application logs) reliably into HDFS for storage and analysis.Apache Oozie in the Hadoop ecosystem is used for:
✔ Correct: C — Workflow scheduling
Oozie = Hadoop workflow scheduler. Manages and coordinates multiple Hadoop jobs (MapReduce, Hive, Pig) to run in the correct sequence and at the right times. Think of it as a cron job manager for Hadoop.The NameNode in HDFS is responsible for:
✔ Correct: A — Metadata storage
NameNode = master node of HDFS. Stores metadata (file names, locations of blocks, access permissions). Does NOT store actual data. DataNodes store the actual data blocks and report to NameNode.The default block size in Hadoop 2.x (HDFS) is:
✔ Correct: B — 128 MB
Default HDFS block size = 128 MB in Hadoop 2.x (was 64 MB in Hadoop 1.x). Large block size reduces metadata overhead. Each block is replicated to 3 DataNodes by default for fault tolerance.Which of the following is NOT part of the core Hadoop ecosystem?
✔ Correct: C — Cassandra is NOT a Hadoop component
Cassandra is a separate NoSQL database (by Apache, developed at Facebook). It is NOT part of the Hadoop ecosystem. Hadoop components: HDFS, MapReduce, YARN, Hive, Pig, HBase, Sqoop, Flume, Oozie, Mahout.Which Hadoop component is a NoSQL database?
✔ Correct: A — HBase
HBase = NoSQL, column-oriented database built on top of HDFS. Provides real-time read/write access to large tables with billions of rows. Hive = query tool (not a database). Sqoop = data transfer. Flume = log collector.MapReduce processing involves which TWO main phases?
✔ Correct: A — Map and Reduce
MapReduce: Map phase processes input data into intermediate key-value pairs. Shuffle & Sort groups by key. Reduce phase aggregates key-value groups into final output. All runs in parallel across cluster nodes.Apache Mahout in the Hadoop ecosystem is used for:
✔ Correct: C — Machine Learning on Hadoop
Mahout = open-source library providing scalable ML algorithms (clustering, classification, collaborative filtering/recommendations) that run on Hadoop. Like having a machine learning toolkit on top of Big Data.Apache Pig in Hadoop is used for:
✔ Correct: B — ETL and data transformation
Pig uses Pig Latin scripting language for data flow and ETL operations. Good for transforming raw data into formats suitable for analysis. Hive = SQL-like queries. Pig = data transformation pipelines.CH.3Spark, NoSQL, Streaming & ToolsQ.27–Q.38
Apache Spark is faster than Hadoop MapReduce mainly because:
✔ Correct: B — In-memory processing
Spark’s key advantage = keeps intermediate data in RAM (in-memory) instead of writing to disk between each step. MapReduce writes to HDFS at every intermediate step — very slow due to disk I/O. Spark can be up to 100x faster.Apache Spark is primarily written in which language?
✔ Correct: C — Scala
Spark = Scala (primary language). But Spark also supports Python (PySpark), Java, and R — making it versatile. Hadoop = Java. This Spark(Scala) vs Hadoop(Java) distinction is frequently tested.Apache Kafka is used for:
✔ Correct: A — Real-time data streaming and messaging
Apache Kafka = distributed event streaming platform. Used as a high-throughput, fault-tolerant message queue for real-time data pipelines. Data producers publish to Kafka topics; consumers subscribe. Used for UPI transaction streams, log aggregation.Which is an example of BATCH processing in Big Data?
✔ Correct: A — Hadoop MapReduce overnight reports
Batch processing = processes data in large scheduled batches (not in real-time). MapReduce is the classic batch tool. Overnight risk reports, monthly payroll, end-of-day reconciliation are batch. Kafka/Flink/Spark Streaming = real-time.NoSQL databases scale primarily by:
✔ Correct: B — Horizontal scaling
NoSQL = Horizontal scaling — add more commodity servers to handle more data/traffic. RDBMS = Vertical scaling (upgrade existing machine). Horizontal scaling is cheaper and virtually unlimited — key Big Data advantage.The CAP theorem in distributed systems states:
✔ Correct: A
CAP Theorem: Consistency (all nodes see same data), Availability (always responds), Partition Tolerance (works despite network failure). A distributed system can guarantee only 2 of the 3 simultaneously.Which of the following is a NoSQL database?
✔ Correct: C — MongoDB
MongoDB = document-oriented NoSQL database (stores JSON-like documents). MySQL, Oracle, PostgreSQL = relational (SQL) databases. Other NoSQL examples: Cassandra, HBase, Redis, CouchDB, DynamoDB.ETL in Big Data stands for:
✔ Correct: A — Extract, Transform, Load
ETL = the process of moving data from source systems into analytics platforms: Extract (pull from sources), Transform (clean, format, enrich), Load (put into data warehouse/lake). Apache Pig is Hadoop’s ETL tool.A Data Lake is best described as:
✔ Correct: A — Raw data in native format
Data Lake = stores all data (structured + unstructured) in raw native format. No schema required upfront (schema-on-read). Data Warehouse = stores processed, structured, query-ready data. Poorly governed Data Lake = “Data Swamp.”Which tool is primarily used for Big Data VISUALISATION?
✔ Correct: A — Tableau / Power BI
Tableau and Power BI (Microsoft) are data visualisation tools that create interactive dashboards and charts from Big Data. Flume = log collection. Sqoop = data transfer. Oozie = job scheduling.Which AWS service is a fully managed cloud data warehouse?
✔ Correct: A — Amazon Redshift
Amazon Redshift = AWS’s fully managed cloud data warehouse for fast analytics on large datasets. AWS EMR = managed Hadoop/Spark. AWS S3 = object storage. Azure BigData = Azure Synapse. Google = BigQuery.Which Google Cloud service handles Big Data SQL queries at petabyte scale?
✔ Correct: A — Google BigQuery
BigQuery = serverless, scalable data warehouse by Google Cloud. Run SQL queries across petabytes of data in seconds. Serverless = no infrastructure management needed. Google Dataproc = managed Hadoop/Spark.CH.4Banking, India & SecurityQ.39–Q.50
Banks primarily use Big Data for which purposes?
✔ Correct: C
Banks use Big Data for: fraud detection (real-time), credit risk scoring, customer segmentation, AML compliance, NPA prediction, personalised offers, and regulatory reporting.For real-time fraud detection in banking, which combination is MOST effective?
✔ Correct: A — Spark Streaming + ML model
Real-time fraud detection requires low-latency stream processing + intelligent scoring. Spark Streaming processes transactions as they arrive; ML model scores each transaction for fraud probability within milliseconds.Customer segmentation in banking using Big Data is primarily done with:
✔ Correct: A — Clustering algorithms
Customer segmentation uses unsupervised ML (K-Means clustering) to group customers by behaviour, demographics, risk profile. Enables targeted marketing, personalised products, and cross-selling without manual classification.Big Data helps in credit scoring for financial inclusion by:
✔ Correct: B — Alternate data for broader credit assessment
Big Data enables alternate data in credit scoring — mobile usage patterns, utility payment regularity, digital transaction history, social media presence. This enables credit assessment for thin-file customers with no credit history (financial inclusion).SEBI uses Big Data analytics primarily for:
✔ Correct: B — Market surveillance
SEBI uses Big Data for market surveillance — detecting abnormal trading volumes, unusual price movements, suspicious order patterns that may indicate insider trading, front-running, or market manipulation in real time.India’s NDAP (National Data and Analytics Platform) is launched by:
✔ Correct: A — NITI Aayog
NDAP (National Data and Analytics Platform) = launched by NITI Aayog as a unified portal for government data discovery, access, and analytics. Makes public datasets easily accessible and standardised.MeghRaj (GI Cloud) is India’s:
✔ Correct: B — Government cloud infrastructure
MeghRaj (GI Cloud = Government of India Cloud) = cloud computing infrastructure for government agencies. Provides scalable, cost-effective data storage and processing for government Big Data needs. By MeitY.UIDAI’s Aadhaar system is significant in the context of Big Data because:
✔ Correct: B — World’s largest biometric Big Data project
UIDAI’s Aadhaar = one of the world’s largest Big Data systems. Stores biometric data (fingerprints, iris scans) + demographic data for 1.4 billion Indians. Demonstrates India’s Big Data capabilities at scale.Data Governance in Big Data ensures:
✔ Correct: B — Quality, integrity, security, compliance
Data Governance = framework of policies, roles, and processes for managing data quality, access control, privacy, and regulatory compliance. Critical for banks — RBI’s IT Framework (2017) mandates data governance.Data Lineage in Big Data means:
✔ Correct: B — Tracking data origin and movement
Data Lineage = tracking the complete journey of data — where it came from, who transformed it, when, and how. Critical for regulatory audit trails (banks must show regulators where their credit risk data came from).Data Anonymisation in the context of Big Data means:
✔ Correct: B — Removing/masking personal identifiers
Anonymisation = permanently removing or masking PII (Personally Identifiable Information) from datasets so individuals cannot be identified. Banks use anonymised data for analytics without exposing customer privacy. Required by DPDP Act 2023.Which of the following BEST describes the main goal of Big Data analytics?
✔ Correct: C — Extract actionable insights for better decisions
The ultimate goal of Big Data analytics = extract meaningful, actionable insights from massive, complex datasets to drive better business decisions. In banking: detect fraud earlier, price risk better, serve customers more personally, comply with regulations more efficiently.⚡
Last-Minute PrepQuick Revision Flash Cards
📊 What is Big Data?
- Huge, complex, fast-growing data
- Cannot be processed by traditional tools
- Needs Hadoop, Spark, NoSQL
- 3 Vs (original) → 5 Vs (extended)
5️⃣ The 5 Vs
- Volume = size/amount of data
- Velocity = speed of generation
- Variety = types (structured/semi/unstructured)
- Veracity = accuracy/trustworthiness
- Value = business benefit
🐘 Hadoop Core
- HDFS = distributed storage (128 MB blocks, 3x replication)
- MapReduce = parallel batch processing
- YARN = resource management
- Developed at Yahoo! Written in Java
🐘 Hadoop Ecosystem Tools
- Hive = SQL-like queries (HiveQL)
- Pig = ETL / data transformation
- HBase = NoSQL database on HDFS
- Sqoop = RDBMS ↔ Hadoop transfer
- Flume = log data collection
- Oozie = workflow scheduler
⚡ Apache Spark
- In-memory processing → 100x faster than MapReduce
- Written in Scala (supports Python, Java, R)
- Batch + Stream + ML (MLlib) + Graph (GraphX)
- Spark Streaming = near real-time
🗄️ NoSQL
- Not Only SQL — handles unstructured data
- Horizontal scaling (add machines)
- MongoDB (document), Cassandra (column), Redis (key-value)
- CAP theorem = choose only 2 of 3
📈 Analytics Types
- Descriptive = What happened?
- Diagnostic = Why?
- Predictive = What will happen?
- Prescriptive = What should be done?
🏦 Banking Uses
- Fraud detection (real-time)
- Credit scoring (alternate data)
- AML / compliance monitoring
- Customer segmentation
- NPA prediction
- Personalised product offers
🇮🇳 India Initiatives
- NDAP = NITI Aayog data platform
- MeghRaj = Govt cloud (GI Cloud, MeitY)
- UIDAI Aadhaar = world’s largest biometric Big Data
- IndiaAI = ₹10,372 crore (2024)
☁️ Cloud Big Data
- AWS: EMR (Hadoop), Redshift (warehouse), S3 (storage)
- Azure: HDInsight, Synapse Analytics
- Google: BigQuery (SQL at petabyte scale), Dataproc
⚡ Streaming Tools
- Kafka = real-time streaming/messaging
- Spark Streaming = near real-time (micro-batch)
- Apache Flink = true real-time stream
- Hadoop MapReduce = batch only
🔐 Security & Governance
- Data Governance = policies for quality + compliance
- Data Lineage = track origin and movement
- Anonymisation = mask PII data
- ETL = Extract, Transform, Load
- Data Lake = raw | Data Warehouse = processed
📌 Must-Know Keywords
5 VsHDFSMapReduceYARN
Hive (SQL)Pig (ETL)HBase (NoSQL)
SqoopFlumeOozie
Apache SparkKafkaSpark Streaming
NDAPMeghRajAadhaar Big Data
CAP TheoremData LakeETL