1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
(env) laptop-124:botoform russellballestrini$ bf --region us-east-1 cli stage

You are now connected to vpc-xxxxxxx (stage) in us-east-1

You now have access to the evpc object, for example: evpc.roles

>>> test1 = evpc.boto.ec2_client.describe_spot_price_history(
        InstanceTypes=["c5.2xlarge", "c4.2xlarge"],
        AvailabilityZone="us-east-1a",
        Filters=[{"Name" : "product-description", "Values" : ["Linux/UNIX"]}])["SpotPriceHistory"]

>>> test2 = evpc.boto.ec2_client.describe_spot_price_history(
        InstanceTypes=["c5.2xlarge", "c4.2xlarge"],
        AvailabilityZone="us-east-1a",
        Filters=[{"Name" : "product-description", "Values" : ["Linux/UNIX (Amazon VPC)"]}])["SpotPriceHistory"]

>>> len(test1)
0

>>> len(test2)
326


>>> test3 = evpc.boto.ec2_client.describe_spot_price_history(
        InstanceTypes=["m4.2xlarge"],
        AvailabilityZone="us-east-1a",
        Filters=[{"Name" : "product-description", "Values" : ["Linux/UNIX"]}])["SpotPriceHistory"]

>>> len(test3)
0

>>> # older generations have prices in both product descriptions, for example `m3`

>>> test4 = evpc.boto.ec2_client.describe_spot_price_history(
        InstanceTypes=["m3.2xlarge"],
        AvailabilityZone="us-east-1a",
        Filters=[{"Name" : "product-description", "Values" : ["Linux/UNIX"]}])["SpotPriceHistory"]

>>> len(test4)
118

>>> test5 = evpc.boto.ec2_client.describe_spot_price_history(
       InstanceTypes=["m3.2xlarge"],
       AvailabilityZone="us-east-1a",
       Filters=[{"Name" : "product-description", "Values" : ["Linux/UNIX (Amazon VPC)"]}])["SpotPriceHistory"]

>>> len(test5)
225



"""
The pricing must be messed up internally because even when we had c4's they almost always got replaced with c3/m3
Russell Ballestrini
@russellballestrini
Feb 06 12:09
We also learned the hard way that if the ASG LC uses ebs optimized, the cost for a spot instance is much more expensive on c3 versus c4
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html <- there is a chart here which basically is free EBS Optimized for anything set to "default"
Cristian Măgherușan-Stanciu @magheru_san
@cristim
Feb 06 12:38
The spot product is currently hard-coded to Linux/Unix, this can explain the weird prices you saw
We have logic that should be able to consider the EBS surcharge
Russell Ballestrini
@russellballestrini
Feb 06 12:39
We use linux exclusively.
Cristian Măgherușan-Stanciu @magheru_san
@cristim
Feb 06 12:40
But do you also have the Linux/Unix VPC spot product?
Russell Ballestrini
@russellballestrini
Feb 06 12:40
I don't know what that is I suppose. Do you have a link?
Cristian Măgherușan-Stanciu @magheru_san
@cristim
Feb 06 12:41
not at the moment
Russell Ballestrini
@russellballestrini
Feb 06 12:41
We use VPC exclusively too, if that is what you mean
Cristian Măgherușan-Stanciu @magheru_san
@cristim
Feb 06 12:41
You can see it in the spot request
Russell Ballestrini
@russellballestrini
Feb 06 12:42
cristim/autospotting#106
this related?
Cristian Măgherușan-Stanciu @magheru_san
@cristim
Feb 06 12:42
Exactly
Russell Ballestrini
@russellballestrini
Feb 06 12:49
Product description Linux/UNIX
is that the spot product?
Russell Ballestrini
@russellballestrini
Feb 06 12:55
I told it to ignore {c3,m3,r3}.2xlarge and it choose to launch a c3.4xlarge over a c4.2xlarge or c5.2xlarge - heh
Cristian Măgherușan-Stanciu @magheru_san
@cristim
Feb 06 12:59
Is it maybe because of the memory sizes?
Russell Ballestrini
@russellballestrini
Feb 06 13:00
c5 has an extra 1G but I would think C5 would trade with a C4 before it traded with a C3
both c3 and c4 have 14G, c5 has 15G
Russell Ballestrini
@russellballestrini
Feb 06 13:14
I'd be glad to help test or debug if you have time and any ideas.
Cristian Măgherușan-Stanciu @magheru_san
@cristim
Feb 06 13:19
Unfortunately I don't have much time these days
The memory seems to explain it
Compare the price of the c3/c4
But yeah, the logging should be more clear about where is the comparison failing
For each of the considered instance types
Patches welcome
xlr-8
@xlr-8
Feb 06 14:19
@russellballestrini just in case you wanted to open an issue/provide more logs - it might get a little too verbose, but that might help dig a bit: https://github.com/cristim/autospotting/blob/master/START.md#debug-autospotting
Russell Ballestrini
@russellballestrini
Feb 07 18:46
@xlr-8 thanks that actually helped a lot
So it seems c5.2xlarge Spot price: 0 Instance price: 0.34
And then looking here (https://github.com/cristim/autospotting/blob/master/core/instance.go#L162) I see that if Spot price is 0 it is not considered for replacement.
So I guess the question is, why is it 0?
Russell Ballestrini
@russellballestrini
Feb 07 19:13
https://github.com/cristim/autospotting/issues/206#issuecomment-363988874
Nathan Sullivan
@CpuID
Feb 07 19:15
definitely doesn't look to be an issue of "no spot price" upstream, can see it in us-east-1 as having a non-zero value for c5.2xlarge instances...
Russell Ballestrini
@russellballestrini
Feb 07 19:17
c4 appears to have the same issue, 0
xlr-8
@xlr-8
Feb 07 23:35
@russellballestrini glad it helped! Indeed the c5 prices should have been added cristim/ec2-instances-info@f464d91 - I'm not sure if c5 instances are globally available, in which region are you running?
Nevermind, I got my answer cristim/autospotting#213
Cristian Măgherușan-Stanciu @magheru_san
@cristim
00:44
it could be because of using an incorrect hardcoded spot product when retrieving spot prices
sheldon-sminq
@sheldon-sminq
02:00
we have been successfully using autospotting for our servers. We have 3 servers under loadbalancer, 1 is on-demand and 2 are spot instances created by autospotting. Today we noticed one issue that our load balancer now has 3 spot instances and the on-demand instance is just running as a stand alone server. Is this a configuration issue?
xlr-8
@xlr-8
02:01
hello @sheldon-sminq hard to say without more information, what options are you currently using?
sheldon-sminq
@sheldon-sminq
02:02
@xlr-8 this is the current parameters set for autospotting AllowedInstanceTypes t2.small,t2.medium
BiddingPolicy aggressive
DisallowedInstanceTypes
ExecutionFrequency rate(5 minutes)
LambdaHandlerFunction handler.Handle
LambdaMemorySize 256
LambdaS3Bucket cloudprowess
LambdaZipPath nightly/lambda.zip
LogRetentionPeriod 7
MinOnDemandNumber 1
MinOnDemandPercentage 17.0
OnDemandPriceMultiplier 1.0
Regions ap-south*
SpotPricePercentageBuffer 10.0
We have an ECS cluster running 3 instances under a load balancer
which has the tag spot-enabled=true
xlr-8
@xlr-8
02:04
You are not running the latest code are you? The Handler name seems like the old one.
I'd suggest removing the MinOnDemandPercentage, as this is ignored as you se MinOnDemandPercentage
If you could open an issue and provide some logs from cloudwatch, that would be helpful - as from what you describe the configuration should be OK. The on-demand server should remain in the ELB. I'm not using ECS, so I've less experience with that type of setting
sheldon-sminq
@sheldon-sminq
02:06
let get a copy of the logs, also how do i upgrade to the latest code?
do i just update cloudformation stack and change the LambdaZipPath?
Cristian Măgherușan-Stanciu @magheru_san
@cristim
02:08
we recently changed the Lambda runtime, you need to change also the handler parameter
xlr-8
@xlr-8
02:08
see here for more information: https://github.com/cristim/autospotting/pull/207/files
Cristian Măgherușan-Stanciu @magheru_san
@cristim
02:10
this is more concise: https://github.com/cristim/autospotting/blob/master/cloudformation/stacks/AutoSpotting/parameters.yaml
xlr-8
@xlr-8
02:10
Indeed :D
sheldon-sminq
@sheldon-sminq
02:15
thanks @cristim @xlr-8 will get back with the logs
xlr-8
@xlr-8
02:17
:+1:
sheldon-sminq
@sheldon-sminq
02:59
Bad handler 'autospotting': need more than 1 value to unpack getting this error
sheldon-sminq
@sheldon-sminq
03:25
re installed the stack works now
Russell Ballestrini
@russellballestrini
05:24
@cristim how is the spot product derived?
Additionally is 'Linux/UNIX' the same as 'Linux/UNIX (Amazon VPC)'?
Cristian Măgherușan-Stanciu @magheru_san
@cristim
06:05
it's not
Russell Ballestrini
@russellballestrini
06:06
Ok, so I think that is my issue. How is the spot product derived?
Cristian Măgherușan-Stanciu @magheru_san
@cristim
06:06
we currently have it hardcoded to 'Linux/UNIX' since the beginning of the project, which has been a constant source of problems
this needs to be made configurable
or detected from the group level
Russell Ballestrini
@russellballestrini
06:07
It is derived from the ASG group or ASG Launch Config?
Cristian Măgherușan-Stanciu @magheru_san
@cristim
06:07
the only way know so far to detect it is by creating a spot request with the AMI used in the launch configuration of the group
the product will be set on the spot request object, which can be cancelled immediately afterwards
Russell Ballestrini
@russellballestrini
06:09
The spots which actually launch are 'Linux/UNIX' but it looks like the pricing data for c4 and c5 (and almost all of the latest 2 generation of instance types) is 'Linux/UNIX (Amazon VPC)'
Cristian Măgherușan-Stanciu @magheru_san
@cristim
06:09
we can even request it with the minimum allowed price of 0.001 so that it won't launch any instance
Russell Ballestrini
@russellballestrini
06:10
I'm not sure it is a detection issue, I think my ASG would work with either products
Cristian Măgherușan-Stanciu @magheru_san
@cristim
06:10
the price is not the same
Russell Ballestrini
@russellballestrini
06:10
Right, but I'm not sure why that would stop us from checking both prices.
Cristian Măgherușan-Stanciu @magheru_san
@cristim
06:11
the code just gets the price for one of these products
it will need to be extended to cover all of them and make it work with detection
Russell Ballestrini
@russellballestrini
06:17
So I just launched a spot and I get Product description Linux/UNIX
And it's a c4.2xlarge so I think it's a mismatch in our expectations.
When we ask for Linux/UNIX prices with autospotting we seem to end up with 0
Cristian Măgherușan-Stanciu @magheru_san
@cristim
06:19
that's strange
but that's likely where the 0 comes from
Russell Ballestrini
@russellballestrini
06:20
I agree but I have no idea why.
Cristian Măgherușan-Stanciu @magheru_san
@cristim
06:20
can you make a support request for this to AWS?
Russell Ballestrini
@russellballestrini
06:21
When I have a better understanding of the issue I can
Cristian Măgherușan-Stanciu @magheru_san
@cristim
06:21
try to get the prices with the aws cli
the support will anyway expect you to use that
@russellballestrini please keep me posted with your progress
Russell Ballestrini
@russellballestrini
06:28
http://pad.yohdah.com/598/spot-product-no-history
^ @cristim
As you can see there is no pricing data for regular ole Linux/UNIX for the latest 2 generations. (and I'm pretty sure this is true for all instance classes)
Russell Ballestrini
@russellballestrini
06:35
but simply changing the product-description to Linux/UNIX (Amazon VPC) gives me the expected pricing data.
This makes sense for c5 because it is mandatory that it needs to launch into a VPC
Cristian Măgherușan-Stanciu @magheru_san
@cristim
06:40
We should request prices for all products we need
"""