- Data structures
Senior Software Developer with Bachelor Degree in Computer Science, Computer Information Systems, Information Technology, or a combination of education and experience equating to the U.S. equivalent of a Bachelor's degree in one of the aforementioned subjects.
Job Duties and Responsibilities:
Deploy a Hadoop cluster, maintaining a Hadoop cluster, adding and removing nodes using cluster monitoring tools like Ganglia Nagios Cloudera Manager.
Configure the NameNode high availability and keeping a track of all the running Hadoop jobs.
Manage and maintain the Hadoop clusters for uninterrupted job.
Routine check-up, back-up and monitoring of the entire system.
Ensure the connectivity and network is always up and running.
Plan for capacity upgrading, downsizing as and when the need arises.
Manage the HDFS and ensuring it is working optimally at all times.
Secure the Hadoop cluster in a foolproof manner is paramount.
Regulate the administration rights depending on job profile of users.
Add new users over time and discarding redundant users smoothly.
Fine tune applications and systems for high performance and higher volume throughput.
Pre-Process using Hive and Pig.
Troubleshoot and develop Hive, Pig, Flume, Mango DB, Sqoop, Zookeeper, Spark, MapReduce2, YARN, HBase, Kafka and Strom.
Translate load and exhibit unrelated data sets in various formats and sources like JSON, text files, kafka queues and log data.
Install and configure Docker images for Telegraf, InfluxDB, Grafana, Kapacitor on AWS cloud monitoring EC2.
Design and Develop Kapcitor scripts for alerting as push notifications, SMS, Email and Slack alerts.
Architect, Design and Develop Big Data streaming applications to use high performance and highly available NoSQL Key Value store Redis for check pointing.
Design and develop AWS Cloud deployment scripts using AWS Cloud Formation Templates, Terraform and Ansible.
Design and Develop Spark applications in Scala that use DOM/SAX parsers for parsing incoming raw string/XML data.
Develop and deploy Chef scripts on centralized DEV, QA, PROD servers for installing Java 8, Apache Spark/Flink/Apex, Stunnel, nginx http server, telegraf agent, Apache Zookeeper on Standalone AWS cloud clusters.
Work experience / Technologies required for the position:
Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
Experience building and optimizing big data data pipelines, architectures and data sets.
Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
Strong analytic skills related to working with unstructured datasets.
Build processes supporting data transformation, data structures, metadata, dependency and workload management.
A successful history of manipulating, processing and extracting value from large disconnected datasets.
Working knowledge of message queuing, stream processing, and highly scalable big data data stores.
Strong project management and organizational skills.
Experience supporting and working with cross-functional teams in a dynamic environment.
At least with 2+ years of experience in a Data Engineer or similar roles, who has attained a Bachelor's degree in Engineering, Computer Science, Information Technology, related field or equivalent work experience.
Experience with big data tools: Hadoop, Spark, Kafka, etc.
Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.
Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.
Experience with AWS cloud services: EC2, EMR, RDS, Redshift
Experience with stream-processing systems: Storm, Spark-Streaming, etc.
5+ years of experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc.
Work location is Portland, ME with required travel to client locations throughout USA.
Rite Pros is an equal opportunity employer (EOE).
Please Mail Resumes to:
Rite Pros, Inc.
415 Congress St, Suite # 201 & 202
Portland, ME 04101