performance - Am I using Julia right? -

i started using julia couple of months ago, deciding give try after weeks of hearing people praise various features of it. more learned it, more liked style, merging ease of expressing concepts in high-level language focus on speed , usability. implemented model i'd written in c++ , r in julia, , found julia version ran faster r version, still slower c++. code more legible in julia in both other languages. worth lot, , in particular i've generalized model, amount of work broaden scope of julia code far less comparable amount of worked threatened in other languages.

lately, have been focused on getting julia code run faster, because need run model trillions of times. in doing i've been guided @code_warntype, @time, @profile , profileview , track-allocation flag. great. toolkit isn't nice other languages' profiling tooling, still pointed out lot of bottlenecks.

what find have in code precisely high-level expressivity in julia, , when rewrite expressivity avoid unnecessary allocations, lose expressivity. trivial example, changed line of code read

sum([x*y x in some_list, y in similar_list]) 

to loop iterates on lists , adds state variable. not rocket science, , understand why it's little faster not have allocate array. it's lot faster though. i've done similar things avoiding using dicts or composite types "feel" right problem, when can instead manually keep track of indices in temporary parallel arrays, style of coding abhor evidently runs lot faster when i'm repeating particular operation of creating , briefly using small data structure lots , lots of times.

in general, largely fine because i've taken heart instructions write short methods, higher-level methods compose behavior of own shorter methods don't need "worry" how shorter methods work; shorter ones can clunky read without making core of program clunky read.

but has me wondering if i'm "missing something." if whole point of language (for me not-theory-concerned end user) in part combine speed ease of development/thought/reading/maintenance etc., i.e. writable , usable technical computing language, doesn't mean shouldn't ever spending time thinking fastest way add bunch of numbers, shouldn't rewriting "easy read" or elegant code takes advantage of mappings, filters, , high-level function concepts "clunky" code reinvents wheel express things in terms of low-level keeping track of array indices everywhere? (some part of me expects language intelligence behind design smart enough "figure out" doesn't need allocate new array when write sum([x*y]), , i'm obtuse figure out right words tell language except literally "telling it" whole loop business manually. @ 1 point thought writing @macros "convert" expressed code long faster loopy expressions, figured must thinking problem wrong if i'm trying extend compiler's functions solve straightforward problems, led me write question.)

perhaps answer "if want performant code, pay price no matter what." or put differently, fast code annoyingly unpleasant read loops, arrays, keeping track of indices, etc. on efficient frontier of speed-legibility tradeoff space. if so, that's valid , not therefore think less of julia. i'm wondering if sort of programming style on frontier, or if experience because i'm not programming "well" in language. (by analogy, see question what productive shortcut vim? , accepted , excellent answer op didn't "get" it.) i'm suspicious though i've succeeded in getting language of want, don't "get" something, don't have sense of ask thing fear don't unknown unknown me.

tl;dr: expected best-practice in julia spend lot of time "decomposing" high-level function calls primitive implementations in terms of loops , arrays in order faster performance, or indicative of me not thinking programming in / using language correctly?

i think topic closely in line discussion going on julia-users group does julia solve two-language problem? , cite here 1 paragraph discussion:

@stefan karpinski:

there different styles , levels of writing code in every language. there's highly abstract c++ , there's low-level pointer-chasing c++ that's c. in c there's void*-style of programming dynamically typed without safety. given fact, i'm not sure solving 2 language problem perspective post posing. enforcing 1 style of programming sounds problem me, not solution. on contrary, think 1 of julia's greatest strengths ability accommodate broad range of programming styles , levels.

my own experience in julia programming, shows can fill blank box of modern programming language bring high level features parallel processing, socket server , ... in hand of scientists, engineers , computation gurus , pragmatic programmers want work in efficient, maintainable , readable manner, using all-in-one programming language.
in opinion using julia in right way, julia other languages represents different styles of programming different situations, can optimize bottlenecks speed, , keep other parts more readable. use tools devectorize.jl avoid rewriting issue.