大数据工程应用 数据分析_大数据工程的方式

It’s been quite an adventure since the dawn of my big data engineer career. I started out differently compared to a typical path. Being a fresh graduate, I possessed nothing more than an academic research background and an enthusiastic attitude. But that wasn’t enough, and I’ve had a couple of rough years.

自从我的大数据工程师职业生涯来临以来，这真是一次冒险。 与典型路径相比，我的开始方式有所不同。 作为一名应届毕业生，我只具有学术研究背景和热情的态度。 但这还不够，我经历了几年的艰难时期。

In life, things turn out to be much simpler when you know what to avoid. I didn’t have that luck, but I’ve learned over my mistakes. There’s a saying in my culture that goes: “You don’t get to choose your job, but it picks you instead”. It was somewhat true in my case. I did not choose big data engineering in the first place.

在生活中，当您知道该避免什么时，事情就会变得简单得多。 我没有那么幸运，但是我已经从错误中学到了东西。 在我的文化中有句谚语说： “您没有选择工作，而是选择了您” 。 就我而言，这是正确的。 首先，我没有选择大数据工程。

In life, things turn out to be much simpler when you know what to avoid

在生活中，当您知道要避免什么时，事情变得简单得多

I committed a lot of errors, chose many wrong paths, and put my faith in some wrong hands. But I’m happy where I am today, I’m willing to pour my heart out to let you know how foolish I was before becoming a big data engineer. There’s nothing to brag about, just my story and what you can get out of it.

我犯了很多错误，选择了许多错误的道路，并将我的信念交到了错误的手里。 但是，我对今天的状况感到很高兴，我愿意为成为大数据工程师之前的工作倾尽全力。 没有什么值得吹嘘的，只有我的故事以及您可以从中学到的东西。

Web开发开始 (Web development as a beginning)

I was eager to land a job, any job. I did some computer programming kind of stuff here and there back in college. Being an immigrant, securing a job is my way to settle down in a foreign country. I wasn’t exactly confident about my technical competence nor professional communication skills.

我渴望找到一份工作，任何一份工作。 我在大学里到处都做过一些计算机编程的工作。 作为移民，找到工作是我在国外定居的方式。 我对自己的技术能力或专业的沟通技巧并不十分自信。

Nevertheless, I began my process as everyone else does: I was spamming my job application. I picked up phone calls, suited up, and went for some of the first official interviews of my life. Things were hard. People turned me down rather than call me back. It didn’t demotivate me as it should, but I started to doubt myself.

但是，我像其他所有人一样开始了我的工作：我向工作申请表发送垃圾邮件。 我打了个电话，适应了我的生活，并进行了我人生中的一些首次正式面试。 事情很难。 人们拒绝了我，而不是给我回电话。 它并没有使我动力不足，但我开始怀疑自己。

I was frustrated. I started to expand my job search to the ones that even slightly related to my study. I initially wanted to land a job as a data scientist or machine learning engineer. But with my zero experience on the fields, that sounded like a mission impossible. Hopefully, there was one startup that accepted my application as a web developer. Compared to my peers, I was underpaid, but I didn’t hesitate for a second. I needed a job, and I took what was offered.

我很沮丧。 我开始将求职范围扩大到与我的学习略相关的职位。 我最初想找到一名数据科学家或机器学习工程师的工作。 但是，根据我在战场上的零经验，这听起来像是不可能完成的任务。 希望有一家初创公司接受我的应用程序作为Web开发人员。 与同龄人相比，我的薪水低，但我毫不犹豫。 我需要一份工作，然后接受了提供的工作。

This was my first mistake. Don’t take me wrongly. The job was fine, everyone seemed friendly, and the tasks weren’t complicated. It just didn’t live up to what I expected I would be doing once I got out of school.

这是我的第一个错误。 不要误会我。 工作很好，每个人看起来都很友好，而且任务并不复杂。 刚毕业，就没有达到我的期望。

I thought I would be cracking some fancy algorithms on a machine learning model. In reality, I was building websites that had nothing to do with probability and statistical learning. I have no offense towards the web development career itself. We weren’t just meant for each other.

我以为我会在机器学习模型上破解一些花哨的算法。 实际上，我正在构建与概率和统计学习无关的网站。 我对网络开发事业本身没有任何冒犯。 我们不只是为了彼此。

I started to slack off on the job. Instead of figuring out a way with my superior, I chose something else: looking for another job. It sounds bad I know. I was on the job only for a couple of months, but the monotony of a workday was eating me up. I was scared, shameful, and too baffled to talk to anyone. I was doing some quick interviews behind everyone’s back, and I landed another job.

我开始放松工作。 我没有与上司想办法，而是选择了其他东西：寻找另一份工作。 我知道这听起来很糟糕。 我只工作了几个月，但一个工作日的单调使我吃饱了。 我很害怕，可耻，也很困惑，无法与任何人交谈。 我在每个人的背后进行了一些快速面试，然后我找到了另一份工作。

The new job was about data mining, big data, machine learning, every big keyword that you can think of. Needless to say, I was hyped to start my new job right away. I tried to tell my manager that I want to quit with a straight face. He wasn’t shocked, which shocks me that he didn’t. Anyway, we figured out the paper works in good terms. I got a 2-day leave notice. I couldn’t come up with a reason for a farewell party, since not everyone knew of my presence.

新工作是关于数据挖掘，大数据，机器学习以及您能想到的每个大关键字。 不用说，我被炒作立即开始新工作。 我试图告诉我的经理，我要坦率地辞职。 他没有感到震惊，这让我震惊，他没有。 无论如何，我们认为论文工作得很好。 我有两天的请假通知。 我无法提出告别聚会的理由，因为不是每个人都知道我的存在。

On my last day at my first job, I had to say goodbye to one particular colleague that I was close to. I said: “I had to go”. She was surprised: “What do you mean? Where are you going?”. I’m not good at saying goodbye, so I was being direct: “I’m going to quit the company”. She went from surprise to anger, and then a little pity.

在第一份工作的最后一天，我不得不向一个与我接近的特定同事道别。 我说：“我必须走了”。 她感到惊讶：“你是什么意思？ 你要去哪里？”。 我不擅长说再见，所以我很直接：“我要辞职了”。 她从惊讶变成愤怒，然后有点可惜。

It was harsh. I didn’t properly handle the job in a professional and emotional way. It was due to my ignorance and naivety that altered my decision. My next hope laid on the next promising job. But little did I know that things just started to get worse on me.

太厉害了 我没有以专业和情感的方式正确地处理这份工作。 由于我的无知和天真改变了我的决定。 我的下一个希望寄托在下一个有希望的工作上。 但是我几乎不知道事情开始变得更糟了。

What you can learn:- You don't have to accept any offers just because it's the only choice. You'll end up regret for it most of the time.
- Communication is the key.
- The grass might be greener on the other side, but it's merely a matter of perspective.

机器学习仍然是炒作 (Machine learning is still a hype)

I’ve got no time to regret my quitting decision because I’m thrilled to get on the next adventure. In this new job, they put me on a new project that no one has been working on, technically. These kinds of newish projects could have many problems. They are usually vague, lack of technical specifications and they turn out to suck.

我没有时间后悔自己的决定，因为我很高兴能踏上下一次冒险之旅。 在这项新工作中，他们使我从事了一个新的项目，该项目在技术上从未有人进行过。 这些类型的新项目可能会有很多问题。 它们通常是模糊的，缺乏技术规范，结果很糟。

As for this project of mine, it had all those three. They promised me on data mining and machine learning applications, but there was nothing to apply on. No data, no software architecture, no technical vision. I spent my whole day attending meetings where people argue rather than work. I didn’t see anywhere I can use my technical competences to my current position.

至于我的这个项目，它具有所有这三个。 他们向我保证要进行数据挖掘和机器学习应用程序，但是没有什么可应用的。 没有数据，没有软件架构，没有技术视野。 我整天都在参加人们争论而不是工作的会议。 我没有看到任何地方可以利用我的技术能力来担任目前的职位。

My manager was unbearable. At some point, he gave me irrelevant tasks just to keep me busy because neither did he have any idea where we were heading to. That’s when I realized the majority of companies claimed to change the world with Machine Learning and Big Data don’t actually have the proper expertise on the job.

我的经理无法忍受。 在某个时候，他给了我无关紧要的任务只是为了让我忙，因为他也不知道我们要去哪里。 从那时起，我意识到大多数声称通过机器学习改变世界的公司实际上都没有适当的专业知识。

I stayed in the company for a while. Mostly because I wanted to legalize my stay in the country. I forgot to mention that ever since I began my professional career, I’ve never stopped my self-learning routine. Especially through difficult times, you’d better be prepared. I’ve jumped into every single piece of advice on how to be updated with the newest trend, and which skills I should focus the most.

我在公司呆了一段时间。 主要是因为我想合法化在该国的居留权。 我忘了提到，自从我开始职业生涯以来，我从未停止过自我学习的习惯。 尤其是在困难时期，您最好做好准备。 关于如何使用最新趋势进行更新以及我应该重点关注的技能，我已经提出了每条建议。

My ultimate path was still ambiguous. I enhanced my portfolio in the most generic way as possible. That was a common error for a junior engineer. But it was hard to avoid. You won’t know what you want to do when you haven’t tried every option. Then came the day when my employer and I couldn’t bear each other much longer. We said goodbye, but not on good terms.

我的终极道路仍然模棱两可。 我以最通用的方式增强了我的投资组合。 对于初级工程师来说，这是一个常见的错误。 但是很难避免。 如果没有尝试所有选项，您将不知道要做什么。 然后是我和我的雇主再也不能忍受彼此的日子了。 我们说了再见，但条件不好。

I hate myself until this day. I let some buzzwords got into my way of pursuing a solid engineering career. I’ve made a promise to never judge a book by its cover anymore. For once, luck has been on my side. This time I would enter into a new world: the Big Data landscape.

我讨厌自己直到今天。 在我从事扎实的工程职业的道路上，我加入了一些流行语。 我已承诺不再以封面来评判这本书。 有一次，运气一直站在我这一边。 这次，我将进入一个新世界：大数据领域。

What you can learn:- Don't believe on what you hear companies say, but on what you see they do.
- Buzzwords are real, and don't fall into traps.

我如何在不知道sh * t的情况下对大数据说是 (How I said yes to Big Data without knowing sh*t)

Remember when Richard Branson said:

记得当理查德·布兰森(Richard Branson)说过的话：

If someone offers you an amazing opportunity and you’re not sure you can do it, say yes — then learn how to do it later.

如果有人为您提供了绝佳的机会，但您不确定自己能否做到，请说“是”，然后再学习如何做。

It might sound cliché but it was true for me once. My Big Data’s encounter happened when I got a job offer to become a Big Data developer. They opened up with me about some requirements on the job that I wasn’t qualified for. Luckily they asked me whether I was ready to enter into an uncertain world of Big Data. Things like Scala or Hadoop ecosystem were distant to me, but I said yes. It was not a confident “yes” as in:” Yes I can do it!”, but rather a what-the-hell-could-go-wrong “yes”.

听起来有些陈词滥调，但对我来说曾经是真的。 我获得大公司成为大数据开发人员的工作时，就遇到了我的大数据。 他们向我公开了一些我没有资格胜任的工作要求 。 幸运的是，他们问我是否准备好进入不确定的大数据世界。 像Scala或Hadoop生态系统之类的东西对我来说很遥远，但我说是的。 这不是一个自信的“是”，例如：“是，我可以做到！”，而是一个错误的 “是”。

Then my adventure in the Big Data world commenced. I started out by skimming some books on Hadoop and Spark — two of the biggest concepts on Big Data at that moment. I managed to comprehend their architecture and technical background, but it was tough to grasp the tools without using it. As I waited long enough to learn on the job, I launched myself forwards the mysterious territory of Big Data.

然后，我在大数据世界中的冒险开始了。 我首先浏览了有关Hadoop和Spark的一些书籍，这是当时有关大数据的两个最大概念。 我设法理解了他们的体系结构和技术背景，但是不使用它就很难掌握这些工具。 当我等待足够长的时间来学习这份工作时，我开始着手转发大数据的神秘领域。

At first, I was amazed by how data didn’t behave in an ordinary way as for small data. The rules were particular here in this domain, and I have to get used to it. But as soon as I understood how things work, I’ve realized there was nothing special about Big Data. I told myself: “It is just big. As an engineer, we merely use the tools to process big data as the way it is, no more no less”. God, I was completely wrong back then.

起初，我对数据没有像小数据那样以常规方式表现感到惊讶。 规则在此领域中特别重要，我必须习惯它。 但是，一旦我了解了事情的运作方式，我就意识到大数据并没有什么特别之处。 我对自己说：“它很大。 作为工程师，我们仅使用工具按原样处理大数据，这同样如此。” 天哪，那时候我完全错了。

Allow me to cite another saying from my native language

请允许我用母语说另一句话

A frog at the bottom of a well regards the world by the size of the well’s circumference.

井底的青蛙通过井的周长大小来关注世界。

I was that frog. I thought Big Data is so small that I could understand it all. I forgot that I was at the edges of the Big Data realm, and my journey has barely begun. Big Data engineering is not only about leveraging tools to process data at scale, but also how we handle this precious resource to bring values into business decision making and client-facing presentations.

我就是那只青蛙。 我认为大数据太小了，以至于我无法理解所有内容。 我忘记了自己处于大数据领域的边缘，而我的旅程才刚刚开始。 大数据工程不仅涉及利用工具大规模处理数据，还涉及我们如何利用这一宝贵资源将价值带入业务决策和面向客户的演示中。

What you can learn:- You can start woking professionally on a field without minimal experience, but that doesn't exists everywhere. Be prepared.
- Be humble. What you know today might be obsolete tomorrow.

结论 (Conclusion)

I’ve listed my errors so you won’t commit them again. A professional career path is long, rigid, and somewhat tedious. We progress quicker with experience, and also by avoiding mistakes made by the ones before us. I hope that when it’s your turn, the journey will be as well adventurous, full of challenges, and fruitful at the end. Until then, happy learning!

我已列出我的错误，因此您不会再次提交。 专业的职业道路漫长，僵化且有些乏味。 我们会凭借经验以及避免遇到的人犯的错误而更快地进步。 我希望轮到您了，这个冒险之旅同样充满冒险，充满挑战，并且最终会硕果累累。 在那之前，学习愉快！

My name’s Nam Nguyen, and I write (mostly) about Big Data. Enjoy your reading? Follow me on Medium and Twitter for more updates.

我叫Nam Nguyen，(主要)写有关大数据的文章。 喜欢阅读吗？ 在Medium和Twitter上关注我以获取更多更新。