What we can learn from the Great California Exodus (Sept. 2012)


By turtledove   Follow   Sun, 23 Dec 2012, 3:52pm   9,587 views   148 comments
Watch (2)   Share   Quote   Permalink   Like (1)   Dislike (1)  

I hope this hasn't already been posted. Didn't see this on the site. Thought it was interesting.

http://www.manhattan-institute.org/html/cr_71.htm

Executive Summary

For decades after World War II, California was a destination for Americans in search of a better life. In many people’s minds, it was the state with more jobs, more space, more sunlight, and more opportunity. They voted with their feet, and California grew spectacularly (its population increased by 137 percent between 1960 and 2010). However, this golden age of migration into the state is over. For the past two decades, California has been sending more people to other American states than it receives from them. Since 1990, the state has lost nearly 3.4 million residents through this migration.

This study describes the great ongoing California exodus, using data from the Census, the Internal Revenue Service, the state’s Department of Finance, the Bureau of Labor Statistics, the Federal Housing Finance Agency, and other sources. We map in detail where in California the migrants come from, and where they go when they leave the state. We then analyze the data to determine the likely causes of California’s decline and the lessons that its decline holds for other states.

The data show a pattern of movement over the past decade from California mainly to states in the western and southern U.S.: Texas, Nevada, and Arizona, in that order, are the top magnet states. Oregon, Washington, Colorado, Idaho, and Utah follow. Rounding out the top ten are two southern states: Georgia and South Carolina.

A finer-grained regional analysis reveals that the main current of migration out of California in the past decade has flowed eastward across the Colorado River, reversing the storied passages of the Dust Bowl era. Southern California had about 55 percent of the state’s population in 2000 but accounted for about 65 percent of the net out-migration in the decade that followed. More than 70 percent of the state’s net migration to Texas came from California’s south.

What has caused California’s transformation from a “pull in” to a “push out” state? The data have revealed several crucial drivers. One is chronic economic adversity (in most years, California unemployment is above the national average). Another is density: the Los Angeles and Orange County region now has a population density of 6,999.3 per square mile—well ahead of New York or Chicago. Dense coastal areas are a source of internal migration, as people seek more space in California’s interior, as well as migration to other states. A third factor is state and local governments’ constant fiscal instability, which sends at least two discouraging messages to businesses and individuals. One is that they cannot count on state and local governments to provide essential services—much less, tax breaks or other incentives. Second, chronically out-of-balance budgets can be seen as tax hikes waiting to happen.

The data also reveal the motives that drive individuals and businesses to leave California. One of these, of course, is work. States with low unemployment rates, such as Texas, are drawing people from California, whose rate is above the national average. Taxation also appears to be a factor, especially as it contributes to the business climate and, in turn, jobs. Most of the destination states favored by Californians have lower taxes. States that have gained the most at California’s expense are rated as having better business climates. The data suggest that many cost drivers—taxes, regulations, the high price of housing and commercial real estate, costly electricity, union power, and high labor costs—are prompting businesses to locate outside California, thus helping to drive the exodus.

Population change, along with the migration patterns that shape it, are important indicators of fiscal and political health. Migration choices reveal an important truth: some states understand how to get richer, while others seem to have lost the touch. California is a state in the latter group, but it can be put back on track. All it takes is the political will.

« First     « Previous     Viewing Comments 109-148 of 148     Last »     See most liked comments

  1. drew_eckhardt


    Follow
    Befriend (1)
    2 threads
    590 comments
    Sunnyvale, CA

    109   10:57pm Tue 1 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    thomaswong.1986 says

    drew_eckhardt says

    Big companies recruit fresh graduates from higher ranked schools in an attempt to secure better talent before it gets recruited elsewhere and becomes unavailable

    A myth created more recently by the media. Fact is many tech and especially SV tech companies seek at least 3-5 years experience to be considered. And thats been true for decades prior.

    Sure, tech companies want 3-5 years of experience and many insist on it. I prefer to hire less experienced employees for their third job so my current company deals with fewer of the inevitable mistakes we all make due to inexperience and we can delegate more for less overhead and higher group productivity for not much more money.

    That said big companies hire straight out of school because it may be the only viable way to get the needed body count with quality control. Late stage startups sometimes do too since good experienced people want significant greenfield projects and equity shares which are no longer available so they can't be hired (when willing to settle for similar tasks and equity upside that won't be life changing they can do a lot better at big companies), sub-par experienced employees do negative work (it takes more time to manage them and clean up their mistakes than just doing the work yourself), and good fresh graduates can make positive contributions in small well defined areas today with bigger projects as they learn.

    In between realizing Interesting and viable startups weren't being funded in Boulder, CO and noting it'd be silly to be someplace other than Silicon Valley I remembered that I had a lot of fun at Qualcomm with an employee number in the low 3000s and thought I might do fun big projects at another large company.

    I tried working for Microsoft and Amazon in Washington and both had hired plenty of fresh graduates from decent places, domestic and elsewhere.

    I've read reports of Apple and Google making nice offers to new graduates but couldn't comment on numbers or back that up with personal experience.

  2. drew_eckhardt


    Follow
    Befriend (1)
    2 threads
    590 comments
    Sunnyvale, CA

    110   11:22pm Tue 1 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    Rin says

    This is much more in line with what I'd seen in the northeast corridor. It's a salary, just approaching of that of internal medicine physicians, but clearly not in some exalted strata above them.

    Salary would have averaged just 60% of my total compensation package at my last big employer with RSU value unchanged from my hire date and could have been under 30% on my fourth year due to back-loading of the RSU grant and stock movement.

    Your salary becomes less interesting as you move farther up the corporate ladder. As an extreme example Larry Ellison netted just $1 in salary for 2010-2011 but $70M in other compensation.

    Startups are more complicated - total compensation package current value is less, but you're betting that a significant fraction of a percent at the time of a liquidity event is seven figures worth of life changing long term capital gains .

  3. drew_eckhardt


    Follow
    Befriend (1)
    2 threads
    590 comments
    Sunnyvale, CA

    111   11:28pm Tue 1 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    thomaswong.1986 says

    thats BS... we might have had a shortage of SW eng back in early 80s. But we certainly have a glut of SW eng. Globally.

    While there's no shortage of people who've worked as software engineers, there are not enough decent experienced people to go around.

  4. Peter P


    Follow
    Befriend (4)
    121 threads
    18,570 comments

    112   11:37pm Tue 1 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    There is both a shortage of software "engineers" but there is also a shortage of such jobs.

    How could that be?

    It is a case of skills mismatch caused by having artificial barriers. We will run into more of these problems.

    Opening the border to talents is always good for both employers and employees. If not, employers will try to outsource the jobs. They will come back eventually (because America has the best culture) but it will cause much pain in the mean time.

  5. Peter P


    Follow
    Befriend (4)
    121 threads
    18,570 comments

    113   11:38pm Tue 1 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    Software "engineers" are not real "engineers." At least they are no more real than financial "engineers."

  6. Sam1000


    Follow
    Befriend
    38 comments

    114   11:48pm Tue 1 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    In the Southern California area a Sr. Software Engineer with 10+ yrs experience makes around $110-120k, nothing more! Architects can make upto $150k/yr.

  7. raindoctor


    Follow
    Befriend (1)
    3 threads
    136 comments
    Mountain View, CA

    115   12:00am Wed 2 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    Peter P says

    There is both a shortage of software "engineers" but there is also a shortage of such jobs.

    If you look charitably, it makes sense.

    They want software engineers, whose skill set match like that of Linus Torvalds. Hence, shortage of engineers for such jobs.

    There are many software engineers, who are not like Linus Torvalds. For them, not many jobs available.

    It is similar to how pet dogs are cloned in Korea for $400K. Once your pet dog dies, you can get a nice clone. To get a proper clone, you need to sacrifice many surrogates and prior clones. Check this for dog cloning: http://goo.gl/uDuVt

    Apply that to software engineering, STEM, Arts, etc. We all want to keep the top 5% and get rid of all others. Where should they go to, except working at Hertz car rental, selling insurance, computers, scams, ponzi schemes, working for walmart, McJobs, etc. I don't blame the companies or managers that are looking for top 5%.

    When you look at macro-level, it has different consequences. There are always jobs for top 5%. There is always shortage in such professions. There are always people who can do average job in those fields where shortages exist. Then, we have McJobs, where fierce competition exists.

    How to solve this? Of course, it requires a total rethink of macro economics, human psychology, and public policy. A rethinking that can let people pursue whatever shit they want, instead of cramming everyone through Business degrees or other degrees for non-existent jobs. It also requires euthanizing rentier class as well as creating safety nets. Safety nets maintain a certain floor for aggregate demand.

  8. Kevin


    Follow
    Befriend
    41 threads
    2,655 comments

    116   3:05am Wed 2 Jan 2013   Share   Quote   Permalink   Like (1)   Dislike  

    Peter P says

    There is both a shortage of software "engineers" but there is also a shortage of such jobs.

    How could that be?

    It is a case of skills mismatch caused by having artificial barriers. We will run into more of these problems.

    Opening the border to talents is always good for both employers and employees. If not, employers will try to outsource the jobs. They will come back eventually (because America has the best culture) but it will cause much pain in the mean time.

    There is a shortage of engineering jobs for mediocre engineers. There us no shortage of jobs for the highly competent.

  9. Rin


    Follow
    Befriend
    46 threads
    2,662 comments

    117   3:42am Wed 2 Jan 2013   Share   Quote   Permalink   Like (4)   Dislike  

    Kevin, the avg internal medicine doctor earns $180-$200K, not just the top 5%. And this is nationwide.

    I was offered a *Senior Engineering* position at IBM Global Services, northeast corridor, and that was at $120K. The only person in that team earning $200K was the regional manager. Thus, your salary stories are only about Silicon Valley, a Bay Area phenomena. And that was the last time I'd bothered with IT, before leaving the field altogether for HF.

    Now, for a person capped at $120K in IT, I'm only into my 3rd year of hedge fund work and I'll be finishing this week at near $700K for 2012. Yet, I'm not telling everyone that I'm some great thing because I know how difficult it is, to get into this field and to stick with it.

  10. Kevin


    Follow
    Befriend
    41 threads
    2,655 comments

    118   11:46am Wed 2 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    Yes, doctors make more than engineers. They also require 10 extra years of education and training. Being a physician is hard. Like i said before, software engineering is not the highest paying career that there is.

    IBM GS is *not* a top talent recruiter. they are a mid range consulting company. I can guarantee you that top firms offer more than double that in the northeast. Good engineers don't interview at IBM these days.

    "IT" is not software engineering. Setting up some corporate employee database is not the same thing as designing a multi billion dollar ad bidding system, an operating system, or an internet scale search engine. The former can earn you a decent salary, the latter is what makes you actually care what top tax brackets are.

  11. Rin


    Follow
    Befriend
    46 threads
    2,662 comments

    119   12:45pm Wed 2 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    Kevin says

    Yes, doctors make more than engineers. They also require 10 extra years of education and training. Being a physician is hard. Like i said before, software engineering is not the highest paying career that there is.

    IBM GS is *not* a top talent recruiter. they are a mid range consulting company. I can guarantee you that top firms offer more than double that in the northeast. Good engineers don't interview at IBM these days.

    General Internal Medicine is not hard. It's boring and tedious but pays the bills. Plus, I took the MCAT, scored in the mid-30s and have a 3.6-3.7 GPA in Applied Chemistry/Chemical Engineering, tougher classes than many premed types. That's a near automatic acceptance to 1 or more US AMA MD programs; I'm not afraid of medical school & its alleged difficulties.

    I'm rather certain that you wouldn't be able to outperform me academically but now that I'm financially secure, I don't need to play those games.

    As for IBM GS, they have the big international clients [ i.e. the big projects ] and thus, once again, you're isolating yourself to some top ~5% category in CS persons. And no, only Google in Manhattan offers more than the financial services sectors in the big apple for software engineers. Other jobs in IT (or software engineering: OS, Real-Time rating, etc) do not pay at the $200K range, outside of the Valley. There are those companies in Massachusetts, and most software engineers are not earning over $150K.

  12. Rin


    Follow
    Befriend
    46 threads
    2,662 comments

    120   1:00pm Wed 2 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    Kevin says

    I can guarantee you that top firms offer more than double that in the northeast.

    I know software engineers all over the northeast, none of them are earning $120K * 2 ~ $240K without an equity plan in place.

    I think you're living in your SV bubble.

  13. bob2356


    Follow
    Befriend
    3 threads
    4,477 comments

    121   1:11pm Wed 2 Jan 2013   Share   Quote   Permalink   Like (1)   Dislike  

    It's amazing how many people that are so near the top of the food chain manage to spend so much time posting on PatNet. Especially since in such demanding positions they should be working 100+ hour weeks. I'm pretty much retired and don't find that kind of time. Fantastic time management, very impressive.

  14. Rin


    Follow
    Befriend
    46 threads
    2,662 comments

    122   1:28pm Wed 2 Jan 2013   Share   Quote   Permalink   Like (2)   Dislike  

    bob2356 says

    It's amazing how many people that are so near the top of the food chain manage to spend so much time posting on PatNet.

    Actually, since I'd started work on this hedge fund work, I've had more and more free time. In other words, I don't feel like these big dollars are real work, it's just managing and moving money around.

  15. Kevin


    Follow
    Befriend
    41 threads
    2,655 comments

    123   2:06pm Wed 2 Jan 2013   Share   Quote   Permalink   Like (1)   Dislike  

    Rin says

    Kevin says

    I can guarantee you that top firms offer more than double that in the northeast.

    I know software engineers all over the northeast, none of them are earning $120K * 2 ~ $240K without an equity plan in place.

    I think you're living in your SV bubble.

    Again, I posit that you don't know any good engineers. Please send resumes if you know people that you think are good and aren't making at least $200k (plus equity).

    I don't live in silicon valley!

    There are lots of reasons why being an internal med. is difficult. I have friends in the field and I don't envy their careers in the slightest.

    And IBM global services is a hack consulting firm that hires people that nobody else wants. Anybody who knows how to write a working java program or set up web sphere can get a job there. They are to software engineering what bank tellers are to financial services.

    @Bob: I'm still on Xmas vacation. I just can't abide misinformation.

  16. New Renter


    Follow
    Befriend (3)
    34 threads
    6,270 comments
    San Jose, CA

    124   2:35pm Wed 2 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    bob2356 says

    It's amazing how many people that are so near the top of the food chain manage to spend so much time posting on PatNet. Especially since in such demanding positions they should be working 100+ hour weeks. I'm pretty much retired and don't find that kind of time. Fantastic time management, very impressive.

    I've noticed that as well. How often do those long hours include commute and long lunches.

    AI suppose if the SEC can chalk up surfing for porn anything is fair game...

  17. Rin


    Follow
    Befriend
    46 threads
    2,662 comments

    125   7:37am Thu 3 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    Kevin says

    Please send resumes if you know people that you think are good and aren't making at least $200k (plus equity).

    Sorry, can't turn over my friends' confidentiality to an anonymous forum. All my DoD friends: MITRE, Raytheon, Sandoz, Lockheed, have given a re-sounding NO, to requests.

    Now, here's one in the regular sector, who's only willing to put up stuff, if I don't give specifics about his companies or product lines. His background in Microsoft is typical for New England residents. If you feel that he's an idiot, then I'll pass that info back.

    ===

    Aaron Rossing (an alias)

    ...

    Educational Software Firm, Boston MA
    2008-present

    [ Prop Software #1 ]is a reading and writing aid designed primarily for the educational market in C#. Here's a list of the tasks covered:
    Designed and implemented object model for assignments, consisting of tasks, documents, and people and a web service API to enable this new functionality, as well as access to existing data regarding student/teacher rostering.

    Implemented a direct access layer to the API in C#.

    Implemented a facade layer to that API in C# for the other application developers on the team.

    Designed and implemented a method for pushing analytics data back to our servers asynchronously in C#.

    ...

    Prior MS Partner Company, Rte 495 MA
    2002-2008

    [ Prop Software #2 ] network solution's original product is a peer-to-peer collaboration platform written for MS. Effort was applied to integrate this with Windows.

    Once that was done, implemented some connectivity interfaces between [ Prop Software #2 ] and other Microsoft Office products in C++/COM.

    Prior to that, I had been working on the [ Prop Software #2 ] Int Server, a kind of “super-client” that could be extended to talk to databases and other systems.

    Designed and implemented a new user interface for the server’s management console in C++/COM, reflecting a more enterprise-level application.

    Designed and implemented new screens in C++/COM to enable new functionality and added to the controller layer, to enable the new screens and functionality.

    Prior to that, I designed the middle tier of a web-based management console in ASP.NET for [ Prop Software #2 ] Mngmt Server, which allowed enterprise administrators to issue or revoke [ Prop Software #2 ] accounts to imported lists of users, as well as apply various policies to those accounts.

    Originally, I had been hired as the architect of the company’s website. Designed the integration of [ Prop Software #2 ] login system side-by-side with Microsoft Passport, as well as a set of ASP.NET and ColdFusion page and code templates that other developers used to ensure the site would respect this integration properly. Implemented 3rd party e-commerce solutions with backend account provisioning and Akamai for component synchronization with client.

    ...

    Education: BS Computer Science, Worcester Polytechnic Institute, Worcester MA

  18. Kevin


    Follow
    Befriend
    41 threads
    2,655 comments

    126   9:41am Thu 3 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    Those sound like the typical "heap" resumes that make it to a phone screen here. I see a few hundred a year. Feel free to PM me a contact email and I can hook them up with a hiring manager. We have an office near Boston if that's where they want to work.

  19. Rin


    Follow
    Befriend
    46 threads
    2,662 comments

    127   9:45am Thu 3 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    Kevin says

    Feel free to PM me a contact email and I can hook them up with a hiring manager

    Let me first ascertain his feelings about his current employer and sector. If he's interested in a move (meaning that there's something lacking in the present position), I'll do just that.

  20. drew_eckhardt


    Follow
    Befriend (1)
    2 threads
    590 comments
    Sunnyvale, CA

    128   1:38pm Fri 4 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    Rin says

    Now, here's one in the regular sector, who's only willing to put up stuff, if I don't give specifics about his companies or product lines. His background in Microsoft is typical for New England residents. If you feel that he's an idiot, then I'll pass that info back.

    His resume seems to be about creating glue which is not technically challenging or novel and would net a phone call from at most one of the groups I've worked for over the last 20 years.

    That could just be a side effect of where he's worked and not an inherent aptitude problem. Assuming that's the case he'd gain access to positions which are more fun and potentially better paying by doing some interesting personal (perhaps open source) projects and noting them on his resume. Substantial design and implementation work for academic classes would do too (compiler construction, robotic swarms, distributed systems, machine learning, lots of things are interesting) There's open courseware for things like that which is free and easier to fit into one's schedule than time in a class room.

    His resume does not reflect the attention to software life cycle and process I want but don't expect from people who've just worked at two jobs given a defacto industry standard of "poor". After twice the job count (some places aren't so bad so you should learn something eventually) it'd be a red flag I'd discuss with the candidate if the resume otherwise looked good.

    I've done the sort of stuff that gets people hired into the kinds of positions we're discussing and will enumerate. In resume form I'd be a lot more specific noting keywords for recruiters (my client side code is C++), implementations for engineers (my client and server are multi-threaded, although I often address asynchrony with event driven code and the actor model), and process specifics for managers (I have my group using a test driven development methodology which some might describe as behavior driven development). Each of those classes of resume reader are evaluating it within their frame of reference, and candidates do better helping them to form positive impressions.

    When I'd worked in industry two years (on the fourth year our project was done and shipped overseas for maintenance):

    I designed and implemented much of a highly available cellular base station billing interface which kept track of all the events associated with calls and allowed phone companies to charge people exactly once for each one. It kept working even when one node failed. The hardware costs were less than 1/20th a database using solution considered meeting the same throughput and realtime constraints. Our group was smallest building a base station component, first by a long time to deliver a component which was highly available, and my manager was promoted to director after the project.

    My pieces included the switch-over mechanism, replicated persistent storage for in-process call events, replicated persistent completed call storage, the recovery code for both subsystems, the event driven infrastructure (pthreads were crashing our dual-core Sun boxes out to their ROM monitors; and as a happy side effect I picked up a couple orders of magnitude in performance after getting rid of them) it ran on, and the test infrastructure. There were no problems running many calls in parallel, input and output, etc. I was already partitioning things into multiple processes to limit failure domains (you don't want something like a formatting failure for the billing system to stop call processing), and running separate nodes with slightly different configurations (ex - tree fan-out, sync interval) so they'd hit edge conditions at different times and bugs were less likely to take the system off line.

    Before that for fun and profit I built a few Linux SCSI drivers. Sequential I/O through the file system on my last drivers ran at nearly head rate (.5K * number of sectors per track * RPM) which was twice what I got with the same Seagate Hawks on the $20K Sun boxes I used for the cellular base station project. Neither file system nor buffer cache were my doing although I didn't slow anything down. The last two handled multiple simultaneous I/Os. For the last one I wrote the firmware for the SCSI controller for a shared memory asymmetric multi-processing solution. It worked well and was elegant. I also wrote an assembler for that chip - that was kludgey perl script but it worked and didn't require rebooting into a supported platform to assemble SCSI chip code.

    In school I took the compiler construction elective where I and a partner built a compiler for a useful subset of 'C'. The professor provided a program outline and probably defined APIs for the various components (it has been twenty years). The result was good.

    When I was 18-19 I built the Linux SCSI subsystem. It was my first non-trivial project and completely sucked with unclean interfaces and no reasonable accommodation for asynchrony. I learned from that and did much better on the following projects.

    Highlights moving forward in time:

    Eight years in I was the first technical employee at a startup where I led a dozen engineers and built a clustered block storage appliance. A competitor on the other side of town which was smarter about marketing built a very similar product and had a $360 million liquidity event. I used agile processes for rapid, more predictable, high quality software delivery. We had our corporate E-mail running on a prototype two months after we started. Data remained accessible with enclosure loss. A simultaneous failure could not loose data. It could snapshot volumes in ten milliseconds. Performance was independent of how many snapshots existed. Snapshots could be mirrored asynchronously at a remote site for disaster recovery. Snapshots could be made writeable so customers could try an old version with their applications before rolling back their volume. It met the real time requirements dictated by the Microsoft Exchange benchmark which was relevant to our target market. I stored the iSCSI initiator generated data CRCs with the meta-data for end-to-end integrity verification. I had background scans for latent faults when the system was idle. I used circular buffers for trace level logging which was persisted for all relevant subsystem / node combinations when a failure was detected so we could fix rare problems we could not reproduce.

    I personally designed the synchronization protocols, persistent data structures (the offspring of a log structured file system and database with transactional support and B+ trees for indexing extent / snapshot combinations where every re-write went to a new location so a failed write would not leave data unreadable), and recovery process. I personally implemented the code which kept track of where customer data blocks lived, application keep alive, the event driven infrastructure, messaging code, and most of the test infrastructure.

    My novel inventions included the end-to-end integrity mechanism, B+ tree/log-structured file-system/extra indirect table combination, and diagnostics.

    After twelve years I was building business critical systems software where the code which shipped to customers could also run in a simulated environment with deterministic execution order among the asynchronous parts and model checking which allowed for pseudo-random testing to find things we couldn't think of test cases for (I'd thought of that, but got introduced to my first real-world implementation by a Digital SRC alumni). The guys building Google's lock service had not yet published their _Paxos Made Live_ paper where they suggest distributed systems don't work when you don't do that using a commercial database as an example. Four years later Microsoft Research published their _MODIST: Model Checking of Unmodified Distributed Systems_ paper where they found design problems in each of the three systems they examined. I worked for the startup on the other side of town doing a good job selling their storage appliance and gained first hand experience with the sort of failures you start noticing once you have enough customers to have repeats of events that happen every few thousand years of machine time. I noted that they had SCSI timeouts when snapshots were deleted (tens of seconds delays) and was not happy with the data loss vulnerability (everything on the node) to a disk error in their block mapping tables. I made meta-data operations so fast (up to 10,000 times faster) that the impact of snapshot deletion was no longer measurable even when I replicated the disk structures so they were no longer a single point of failure. My data structure was was inherently resistant to bit rot (Some sectors on magnetic media go bad when other bits are changed nearby. Modern drives notice and repair the degradation on reads, although with cached data and long enough uptimes that might not happen) . That was a novel transactionally consistent structure which mixed new transactions with refresh entries in a circular buffer that guaranteed the previous copy of an entry was re-written before it was over-written. I productized that with an on-line rolling upgrade. The disk error simulation model I built to validate my error handling also found that the existing five year old code was not crash consistent.

  21. drew_eckhardt


    Follow
    Befriend (1)
    2 threads
    590 comments
    Sunnyvale, CA

    129   1:40pm Fri 4 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    At the fourteen year mark I wrote the code which made amazon.com's shopping cart elastic so the cluster could grow during high-use seasons and shrink at other times for lower total cost of ownership. I also worked in the S3 group. While not technically challenging it gave me first hand experience dealing with thousands of nodes.

    After sixteen years I was doing high performance (300,000 writes per second per node) fault tolerant (loosing nodes is not a big deal) strongly consistent object storage (the user can't observe any differences between copies) which might be applied to the financial industry. I designed and implemented the code which automatically moved network addresses around so failures were transparent to clients (distributed lease managers are fun), the replication code, recovery processes which check-pointed and restarted, nearly all the simulated environment that ran on (I had a few off-shore guys to help me on that), and a big pile of automated tests. I also did a circular logging system which emptied into shared memory that survived process termination, had a simple interface, and 300,000,000 entry per second throughput which gave me trace level logging on failures in deployed systems without a significant performance penalty (even 50 log entries per write would be just 5%). My simulation and logging systems allowed me to fix the sorts of bugs that took co-workers three weeks in their code not leveraging that functionality in thirty minutes. I met real time constraints when recovering. I optimized mean time to recover which in turn increased reliability which is ultimately inversely proportional to that.

    Twenty years in I'm responsible for the data path in an enterprise grade on-line backup product. I make the company more valuable by enabling exponential growth and with superior engineering to keep margins high due to reduced costs of goods sold and support burdens. I work with product people and the operational group to give them what they need (as opposed to what they think they need) in the most efficient way. After inheriting a minimum viable product I put some process in place, lots of testing, and error injection resulting in order-of-magnitude availability increases and getting the support burden low enough that my "team" (myself plus one minion) didn't need to hire more people in spite of exponential growth. I did some novel WAN optimizations which cut the load on our servers 85% so our ratios of computing/disk/network resources to customers are lower along with the cost of goods sold. I increased performance substantially (one large customer's backup time went from two days to two hours) so we can handle much bigger and more profitable customers. I implemented other features to facilitate on-boarding larger customers at lower cost (time-of-day determined throttling so we don't need to do a side load with fedexed disk drives because a full speed network upload would interfere with their business and one that was always slow would take quite a while).

    I've also done compilers, debuggers, digital video for broadcast television, and digital evidence processing for the police market. I do whatever else goes with the job - fix bugs in and enhance open source we leveraged because it was the right build vs. buy decision, a little technical writing for documentation and patent specifications, automating builds, pickup whatever language the team prefers for test automation or embedding, etc.

    That's meaty. A lot of the guys I've worked with and hired over the decades do similarly beefy things.

    The big common theme is "share holder value," where the resulting products could be sold to big markets and my processes make things happen sooner with reduced runways implying less dilution and more time to pivot. More valuable results mean more accommodation for costs like salary.

    Next up is "products" which marketing people channeling McKenna and Moore (of Crossing the Chasm) would describe with the term "whole product" which means the core product plus everything else that collectively provide enough value to compel purchase. Software engineers usually like to write software which is good but not enough because it's not a whole product. As a minimum a software whole product must have test coverage and process so it works well enough for internal use or sale and have what's needed for after sale support. Building more of a whole product means less other work is needed and a bigger piece of the pie is available for you. Somewhat surprisingly it also increases the fraction of time developers spend actually writing software which is why we got into the business. It also makes for better schedule predictability with fewer fire drills.

  22. raindoctor


    Follow
    Befriend (1)
    3 threads
    136 comments
    Mountain View, CA

    130   3:38pm Fri 4 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    Drew,

    You should build your hft (high freq trading) platform with the skills you have. If you get good at it, you can make millions by joining GS and the like.

    You have all necessary skills. If you add that financial engg juice, your hft can beat any one out there. Wish I had such programming skills! I am more of a systems admin.

  23. Kevin


    Follow
    Befriend
    41 threads
    2,655 comments

    131   3:55pm Fri 4 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    raindoctor says

    Drew,

    You should build your hft (high freq trading) platform with the skills you have. If you get good at it, you can make millions by joining GS and the like.

    You have all necessary skills. If you add that financial engg juice, your hft can beat any one out there. Wish I had such programming skills! I am more of a systems admin.

    Guys who work on those systems get paid ridiculous amounts of money, but it's not life-improving work. It doesn't make you happy or make the world a better place.

    If making $700k a year instead of $350k a year matters to you, then that's one route to try.

    I have a personal rule which is that I won't work on any product that I don't use myself. While this cuts me off from some lucrative and possibly better for the world opportunities, it keeps me happy and engaged in what I do. I love going to work because I know that something I use every day will get a little bit better.

  24. drew_eckhardt


    Follow
    Befriend (1)
    2 threads
    590 comments
    Sunnyvale, CA

    132   4:36pm Fri 4 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    raindoctor says

    Drew,

    You should build your hft (high freq trading) platform with the skills you have. If you get good at it, you can make millions by joining GS and the like.

    Joining Goldman Sachs would

    1) Require me to move to New York.

    2) Put me in an environment which exceeded my BS threshold. While my crap tolerance is higher than is healthy when I've built a lot of a product and/or group, it has remained low in other situations because some one else will pay me more to work in a better environment.

    I'd be willing to put up with that for a year or two with a $10M salary per annum, after which I'd take the half remaining after taxes, invest it, and use that to pay for my middle class lifestyle while I worked full time bootstrapping my own companies which solved interesting problems and were enjoyable places to work because they got software process right and didn't make business decisions I found obviously bone-headed. If I lasted two years I'd do the same and hire a minion once I had work for one.

    I'm not different enough from engineers willing to work for 1/10th that for Goldman Sachs to take me up on my offer so it won't happen.

    In theory I could just build something and try selling it although that's likely to return zero (these things work much better when you start standing on the shoulders of giants than starting at toe level, and to be successful you really need to know your customer) and I don't feel like splitting my engineering focus between day job and elsewhere (both things would suffer).

    You have all necessary skills. If you add that financial engg juice, your hft can beat any one out there. Wish I had such programming skills! I am more of a systems admin.

    Sanity has its price and I'm willing to pay it.

    My plan to make millions starts with evaluating startups' potential for success and how interesting my work would be. I join ones that will be fun and I don't suspect will fail based on experience, early enough in their life cycle to allow a sufficient equity share. Eventually one will pay off (I'm getting better at spotting losers before I join) at which point I'll bootstrap my own companies without compromising my middle class lifestyle. Until then I have fun working on enjoyable projects with other smart people (wise grey bearded engineers, young CS graduates I can form into software ninjas, even some executive types).

  25. mell


    Follow
    Befriend (7)
    274 threads
    3,475 comments
    San Francisco, CA

    133   6:15pm Fri 4 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    Assuming that Kevin and Drew have similar background/experience/skillset it confirms my other assumption that they are at the top 5%, maybe even among the top 1% of programmers. Sure you've worked with similar peers, but the reality is that the majority of companies don't have access to all highly-skilled resources and even if they pay well the competition for those is hard. So there is a trade-off for simplicity (as in simple to understand for the masses) in the code you write vs how expressive/clumsy your code is, and there are important considerations regarding what techniques/technologies and platforms to use and how to make the systems easily extendable and maintainable for less skilled workers for years to come. Despite having worked with assembly in the 68xxx days, then moving on to C/C++, having worked on garbage collectors, Java Compiler/VMs and a little on the C/C++/Fortran compilers I don't consider myself a pro as the field of HW/SW is immensely vast and IMO it pays to keep a system as simple as possible in all aspects unless there is an absolute necessity to go extra nifty. That's why facebook hasn't moved much away from PHP for such a long time (they seem to have plenty patches and other gimmicks to make it perform better though) and it still works extremely well for them on that massive scale and they can easily recruit new coders.

  26. Rin


    Follow
    Befriend
    46 threads
    2,662 comments

    134   8:56pm Fri 4 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    raindoctor says

    You should build your hft (high freq trading) platform with the skills you have. If you get good at it, you can make millions by joining GS and the like.

    You have all necessary skills. If you add that financial engg juice, your hft can beat any one out there. Wish I had such programming skills! I am more of a systems admin.

    Raindoctor, let me tell you this ... science & engineering individuals, hate to work with management consulting types. This is the main reason why I'd left both, the employment of applied chemistry and IT, for early stage hedge fund work.

    And HFT isn't all that it's cracked up to be, in terms of its hierarchy for many trading firms. A lot of trading is still, discretionary, based upon the preferences of the prop trader(s) in charge. Some HFT is still applied to either scalp the spreads or in my firm's case, to manage the risk portfolio of short term positions. And it's quite helpful in that regard. Some a handful of boutiques use HFT alone, in generating alpha.

    All and all, if it's not your cup of tea, it's better to either stick with programming or go into Patents or medical school.

  27. thomaswong.1986


    Follow
    Befriend
    27 threads
    6,131 comments

    135   10:31pm Fri 4 Jan 2013   Share   Quote   Permalink   Like (1)   Dislike  

    drew_eckhardt says

    While there's no shortage of people who've worked as software engineers, there are not enough decent experienced people to go around.

    there is actually a shortage of good managers.. the little idiots who pose as managers cant train their directs is the bigger issue at hand..

    experience.. you must be kidding.. experience in what.. creating what no one created before..

    now imagine that back in the 70 - 80s .. when no one was qualified to their jobs ?

  28. thomaswong.1986


    Follow
    Befriend
    27 threads
    6,131 comments

    136   10:40pm Fri 4 Jan 2013   Share   Quote   Permalink   Like (1)   Dislike  

    Kevin says

    Yes, doctors make more than engineers.

    you should check with your sales team one day.. they make a heck of alot on commissions selling SW (and/or HW). ERP sales from one deal can be several million dollar with a commission of 5-10% off the top can be a big pay off $250-500K in a single month.

    Forget SW engineers but rather go talk to the salespeople in the field... they will tell you.
    After you figure this out ... might want to think about changing your profession.

  29. mell


    Follow
    Befriend (7)
    274 threads
    3,475 comments
    San Francisco, CA

    137   10:46pm Fri 4 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    thomaswong.1986 says

    there is actually a shortage of good managers..

    Agreed.

  30. JodyChunder


    Follow
    Befriend (2)
    32 threads
    1,776 comments
    Victorville, CA

    138   10:50pm Fri 4 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    Kevin says

    Want to see my W2?

    You should go solo. My engineer daughter makes more than you and she's 1099.

  31. Rin


    Follow
    Befriend
    46 threads
    2,662 comments

    139   10:52pm Fri 4 Jan 2013   Share   Quote   Permalink   Like (1)   Dislike  

    mell says

    Assuming that Kevin and Drew have similar background/experience/skillset it confirms my other assumption that they are at the top 5%, maybe even among the top 1% of programmers. Sure you've worked with similar peers, but the reality is that the majority of companies don't have access to all highly-skilled resources and even if they pay well the competition for those is hard

    There is one thing that I like about their respective attitudes, it reminds a bit of undergrad, where we had these idealistic visions of molecular drug delivery [ ligand or time-based ] and other types of scientific projects, which we could never accomplish by working for corporate America, and their short-term vision of simply *getting the stuff* into pre-clinical trials w/o toxicological effects.

    I presume that this zeal is still possible in software, as information tech is still in its 2nd (or 3rd) phase of development, whereas bulk chemicals and biopharma, are mature and no longer respond to that initial urge of exploration and analysis.

  32. Kevin


    Follow
    Befriend
    41 threads
    2,655 comments

    140   11:49pm Fri 4 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    thomaswong.1986 says

    Kevin says

    Yes, doctors make more than engineers.

    you should check with your sales team one day.. they make a heck of alot on commissions selling SW (and/or HW). ERP sales from one deal can be several million dollar with a commission of 5-10% off the top can be a big pay off $250-500K in a single month.

    Forget SW engineers but rather go talk to the salespeople in the field... they will tell you.

    After you figure this out ... might want to think about changing your profession.

    Some salesmen are very good and make considerable sums of money. As a whole, though, software engineers are going to get paid much more. I am friends with plenty of sales folks here. They do well, but not nearly as well as I do. The top salesmen tend to wind up in senior management of the sales org, and make similar compensation to the VPs in the engineering org. If sales is what they love to do, then good for them.

    I really don't understand why you people think anyone should change their profession just for more money. I absolutely love what I do for a living, and I make more than 99% of Americans doing it. I have more than adequate income to own a fine home, save for retirement, and pretty much do whatever I like.

    I have zero interest in sales and even less in something as soul crushing as HFT. I would probably make considerably less at those things because I have no passion for them.

    Only a moron trades a career that they love for the hope of making a little bit more money. Writing software has been my passion since I was a child. It is who I am, not merely what I do.

    JodyChunder says

    You should go solo. My engineer daughter makes more than you and she's 1099.

    Contracting sucks. You rarely get to work on interesting projects. While the base pay is a bit better than a FTE, you're also getting the shaft on benefits. In a recession, you'll be the first to go, and your income in general varies considerably. Only FTEs work on the cutting edge software projects at interesting places.

    I was a freelance engineer for three years and all I wound up doing was boring work.

    Rin says

    I presume that this zeal is still possible in software, as information tech is still in its 2nd (or 3rd) phase of development, whereas bulk chemicals and biopharma, are mature and no longer respond to that initial urge of exploration and analysis.

    Indeed, the relative youth of software engineering is what makes it still so exciting. However, there are equally exciting opportunities in other technical fields; robotics, biotech, prosthetics, energy, and even transportation to name a few.

    I have a friend who is a well to do manager at Roche who tells me that he went into management because the pay for rank and file researchers wasn't enough to support his family. He has a PhD in organic chemistry and has published more papers than I've read in my life. His day job is managing other people who try to make incremental improvements to medications. I can't imagine my life if I wasn't doing what I love every day.

  33. thomaswong.1986


    Follow
    Befriend
    27 threads
    6,131 comments

    141   12:29am Sat 5 Jan 2013   Share   Quote   Permalink   Like (1)   Dislike  

    Kevin says

    I really don't understand why you people think anyone should change their profession just for more money. I absolutely love what I do for a living, and I make more than 99% of Americans doing it.

    Hey .. if you like it great! But its a myth that SW people make alot of money...
    as a Controller currently for Hi Tech company and former big 8 auditor on Hi Tech accounts there is no proof i seen that SW eng.. make loads of cash. I always say.. go talk to sales people to get the real load down.

  34. JodyChunder


    Follow
    Befriend (2)
    32 threads
    1,776 comments
    Victorville, CA

    142   12:33am Sat 5 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    Kevin says

    I was a freelance engineer for three years and all I wound up doing was boring work.

    You make some valid points, Kevin. FWIW, my daughter didn't do it for the money - she's not a greedy bitchface - she just hated being tethered to a cubicle with all the politics and upward failing that is so common in most workplace environments staffed with entrenched snakes. She's a much happier person now.

  35. JodyChunder


    Follow
    Befriend (2)
    32 threads
    1,776 comments
    Victorville, CA

    143   12:42am Sat 5 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    thomaswong.1986 says

    I absolutely love what I do for a living, and I make more than 99% of Americans doing it.

    You've got more passion than I do. I love land lording, and I love my yoghurt shop and I love love love my dent repair biz - but only in as much as I make so much more $$$ than most stinkers around here and get to enjoy being a Big Dan around town. I'm not Vincent Van Gogh about it or anything...I don't love it so much that I'd replace a headliner in 120 degree heat for nothing!

  36. Rin


    Follow
    Befriend
    46 threads
    2,662 comments

    144   8:35am Sat 5 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    thomaswong.1986 says

    But its a myth that SW people make alot of money...

    as a Controller currently for Hi Tech company and former big 8 auditor on Hi Tech accounts there is no proof i seen that SW eng.. make loads of cash. I

    After having read Kevin's stream of posts on this matter, I think I'm beginning to see his p.o.v. In the arena of trading, this is a type of winner take all scenario. A true prop trader, who uses software to manage his stop-loss but relies more on his intuition to lock onto the high probability trade(s), and does this in a repeatable manner, will never earn less than half a million a year, anywhere he goes. In effect, he becomes his own institution and continually turns down multimillion dollar offers from HFs, all over. My own company has this sort of task, split up among a few of us, as we have no hope of recruiting such an individual, at least for another few more years.

    In Kevin's case and others, it's the type of person who can pretty much offer a complete solution, right from the get-go of a project. And thus, they can command what seems like an over the top engineering salary, because once their vision is in place, they can assign individuated tasks to lesser paid team members to carry out to completion. But as others have stated, I also see this as a top 1-5% job tier, kinda like the job of Sherlock Holmes, whereas much of Britain's law enforcement still has to work for Scotland Yard.

  37. drew_eckhardt


    Follow
    Befriend (1)
    2 threads
    590 comments
    Sunnyvale, CA

    145   12:53pm Sat 5 Jan 2013   Share   Quote   Permalink   Like   Dislike  

    thomaswong.1986 says

    experience.. you must be kidding.. experience in what.. creating what no one created before..

    Yes.

    The fundamentals haven't changed over the last thirty years.

    Good software engineering is still mostly about implementing solutions to new problems through the creative application of existing knowledge with trade-offs making them practical in terms of development, runtime, and support costs. Occasionally going farther along the creativity axis is a big help, although if that was more of the job we'd be computer scientists doing research not engineers building products.

    Our tool boxes get bigger with time as researchers discover new things, programmers find practical solutions, and that knowledge becomes sufficiently disseminated. Bigger, faster, and cheaper computing resources enable larger more complex solutions or at least make them commercially viable.

    None of that changes the fundamentals.

    New developments tend to be evolutionary not revolutionary. The software implementing a storage system needn't be too different for solid state drives running 20,000 operations per second each versus 200 for fast spinning magnetic drives. You could do backup similarly to an internet protocol connected appliance on a local area network and cloud back end. Leslie Lamport is on at least his 11th Paxos paper in 20-30 years (Paxos is a distributed write-once register protocol which becomes interesting when you use an instance for each successive input to a distributed state machine doing something like tracking your bank balance - you want every deposit to count and every withdrawal to be charged just once regardless of what ATMs and bank computers crash during the process).

    There are internal similarities across broad ranges of products. Secondary storage problems for cellular infrastructure, broadcast video, disk blocks, E-commerce shopping carts, or other objects have a lot in common. I'd extend that within classes of software - systems software, distributed systems, N-tier web applications, etc. Most broadly there are problems where the things you should have learned on your way to a computer science degree play prominent roles and ones where they don't (although they may have been to the people who built the components you're re-using). Relevant experience is relevant wherever it comes from.

    Experience doing ancestrally or evolutionarily related products does lead to faster more predictable development schedules, better performing results, and more reliable products with lower support costs but needn't come from identical experience and is less important where the individual doesn't have technical leadership responsibilities in the area.

    now imagine that back in the 70 - 80s .. when no one was qualified to their jobs ?

    There's always something close enough and usually not even a big stretch. Programming a micro-computer in assembler or 'C' isn't too different from doing the same with the mini-computer which preceded it. Linux on an i386 personal computer (c. 1991 onward) is remarkably similar to BSD 4.2 on a VAX 11/780 (c. 1983 onward) in a cold machine room with a raised floor. The BASIC compiler with various RISC targets I worked on for my first job in industry wasn't too different from the subset of 'C' compiler for the VAX I built in school.

  38. Kevin


    Follow
    Befriend
    41 threads
    2,655 comments

    146   10:54pm Sat 5 Jan 2013   Share   Quote   Permalink   Like   Dislike (1)  

    Rin says

    In Kevin's case and others, it's the type of person who can pretty much offer a complete solution, right from the get-go of a project. And thus, they can command what seems like an over the top engineering salary, because once their vision is in place, they can assign individuated tasks to lesser paid team members to carry out to completion. But as others have stated, I also see this as a top 1-5% job tier, kinda like the job of Sherlock Holmes, whereas much of Britain's law enforcement still has to work for Scotland Yard.

    You're grossly overcomplicating things.

    There are, perhaps, 50,000 "great" SW engineers in the US, and perhaps 200,000 or so worldwide.

    There are two types of software firms:

    - Those that succeed by constant innovation
    - Those that are coasting along through legacy contracts and playing in boring markets that somebody has to fill.

    The first group cares about the 50,000 great engineers. The second group is happy with the other 450,000.

    The second group can get engineers at good but not great salaries. Salaries that are respectable and will make for a comfortable living. Solid middle class salaries. Someone in this group with a decent level of experience can take home $120k a year. Not bad in a country where most people make less than $50,000.

    The first group, on the other hand, has to compete aggressively for those 50,000 people. Google, Amazon, Facebook, top-tier trading firms, government organizations, and dozens of startups all vie for their attention. Even second group companies need a few of them now and then.

    We're not running the whole show, we're just very good at what we do, and we get paid accordingly, because our employers are making billions of dollars from the work that we do.

    This isn't really any different from any other industry. There are millions of people who work in financial services, but only a few thousand (maybe not even that many) who are top-tier traders making seven figure salaries. There are more than 700,000 physicians in the US, but only a few hundred are multi-millionaire neuro surgeons.

    What's unique about software engineering is that the gap in pay between "average" and "great" performers pay wise isn't nearly as large as it is elsewhere. A mediocre engineer still gets paid six figures for some reason. It's also one of the very few professions that pays as well as it does without requiring an advanced degree or other specialized training.

  39. thomaswong.1986


    Follow
    Befriend
    27 threads
    6,131 comments

    147   11:02pm Sat 5 Jan 2013   Share   Quote   Permalink   Like   Dislike (1)  

    Kevin says

    What's unique about software engineering is that the gap in pay between "average" and "great" performers pay wise isn't nearly as large as it is elsewhere.

    I dont know how those SW/HW Engineers back in the 80s ever made it to Y2K.

    Guess they were all just average or something, nothing to brag home about.

  40. raindoctor


    Follow
    Befriend (1)
    3 threads
    136 comments
    Mountain View, CA

    148   2:42pm Wed 9 Jan 2013   Share   Quote   Permalink   Like (1)   Dislike  

    Kevin says

    We're not running the whole show, we're just very good at what we do, and we get paid accordingly, because our employers are making billions of dollars from the work that we do.

    I think companies where folks like you work are not paying you enough. They should be paying you a c-level exec salaries. That will keep the average software Joe from great ones.

« First     « Previous comments    

Premium member turtledove is moderator of this thread.

Email

Username

Watch comments by email
Home   Tips and Tricks   Questions or suggestions? Mail p@patrick.net   Thank you for your kind donations

Page took 167 milliseconds to create.